\section{Introduction}
People spend plenty of time indoors such as bedrooms, kitchen, living rooms, and study rooms. Online virtual interior design tools are available to help designers to produce solutions of indoor-redecoration in tens of minutes. However, the online indoor-redecoration in a shorter time is demanding in recent years. One of the bottlenecks is to automatically process the indoor layout of furniture, and the goal is to come up with an assistive model that helps designers produce indoor-redecoration solutions at the industry-level in a few seconds.

Data-hungry models are trained for computer vision and robotic navigation \cite{Dai_2018_CVPR,Gordon_2018_CVPR},  and then two major families of approaches have emerged in the line of auto-design work. The first family of models is object-oriented that the objects in the indoor-scene and their properties are represented \cite{10.1145/2366145.2366154}. In \cite{10.1145/3306346.3322941}, a model is developed to take the advantages of spaces and objects. However, these assistive models are unsatisfactory since they can only satisfy partial requirements from the customers, and the resulting auto-layouts are towards the same kind. Motivated by the challenge, we propose an assistant model that produces different types of auto-layout solutions of indoor-scene for different groups of customers at the industrial end (see Figure \ref{fig1}). Specifically, we propose a conditional auto-layout assistive model where the condition is the representation of the preferences/requirements from customers/property owners. Firstly, the proposed model generates a structural representation according to the conditional representation of different requirements. This conditional generation is produced by a trained conditional model. Secondly, a generated graph and the representation of the condition are mixed through a deep module. Finally, a location module is trained to give the prediction of the location of furniture for different requirements.

\begin{figure}
\centering
\includegraphics[height=7.0cm]{fig1.jpg}
\caption{Hierarchical indoor scenes generation. For example, given an empty tatami room, the layout of objects are produced according to four different requirements from the property owners, including the sleep-functional tatami, the tea-functional tatami, the work-functional tatami and the storage-functional tatami.}
\label{fig1}
\end{figure}

In this paper, we first introduce the related work in Section $2$. The indoor synthesis pipeline which consists of three proposed modules is described in Section $3$. In Section $4$, we propose a graph representation procedure to extract the geometric indoor-scene into the abstract graph. We also propose a dataset of the indoor layout which is collected from the designs of professional interior designers at the industrial end. In Section $5$, a model for a graph generation with conditions is developed where the condition is based on the preferences of the property owners. In Section 6, a conditional scene instantiation module is developed to produce the location of furniture for different property owners. In Section 7, we evaluate the proposed model for three types of rooms including the tatami, the balcony room, and the kitchen. Finally, we discuss the advantage and the future work of the proposed model in Section 8.

\section{Related Work}
We discuss the related work of indoor furniture layout including the indoor-scene representation, graph generative models and the location of furniture models.

\subsection{Indoor-scene Representation}
The study of indoor-scene representation starts two decades ago. In the early years, rule-based formulation is developed to generate the 3D object layouts for pre-specified sets of objects \cite{Xu02constraint-basedautomatic}. Interior design principles are applied through the optimization of the cost function \cite{Merrell:2011:IFL}. The object-object relationships are analyzed in the interior scenes \cite{10.1145/2010324.1964981}. Data-driven models are then developed in this field. The earliest data-driven work is modeled to produce the object co-occurrences through a Bayesian network \cite{10.1145/2366145.2366154}. The followup work make use of the undirected factor graph learned from RGB-D images \cite{article}. Besides, other interior scene representations are applied to models including RGB-D frames, 2D sketches of the scenes, natural language text, and RGB-D representations. However, these representations of the indoor scenes are only practical to a limited range of indoor scenes.

The learning frameworks for the representation of the indoor scene are developed recently. These learning methods apply a range of techniques including human-centric probabilistic grammars \cite{Qi_2018_CVPR}, generative adversarial networks on a matrix representation of the interior scene \cite{10.1145/3303766}, recursive neural networks trained on 3D scene hierarchies \cite{10.1145/3303766}, and convolutional neural networks trained on top-down image representation of interior scenes \cite{10.1145/3306346.3322941}. However, these learning methods are not able to obtain a high-level representation of the indoor scene. Besides, they are not practical at the industrial end as the generated in-door scenes cannot meet the personalized preferences of the property owners.

\subsection{Graph Generative Models}
Another line of recent work on learning the graph-structural representation through deep neural networks is explored. These work aim to learn generative models of graphs. The early work along the line is to develop a graph convolutional neural network (GCN) to perform graph classification \cite{8953909}. The message passing between nodes in a graph \cite{Gilmer2017} is applied to the graph generation of interior scenes \cite{10.1145/3306346.3322941}. However, these graph generative models are not able to produce graphs based on different conditions and meet the personalized preferences of property owners.

\section{Indoor Synthesis Pipeline}
\begin{figure}
\centering
\includegraphics[height=3.0cm]{fig2.jpg}
\caption{The proposed indoor synthesis assistive model for personalized preferences. It consists of graph extraction, conditional graph generation and conditional scene instantiation.}
\label{fig2}
\end{figure}
In this section, we tackle the following indoor-scene layout problem: given the architectural specification of a room (e.g., walls, doors, and windows) of a particular type (e.g., kitchen or tatami), and the personalized preferences of a house owner for the indoor-decoration, we choose and locate a set of objects to decorate that type of a room. We aim to build an assistant model that can support the designer to produce a decoration solution quickly through auto-layout of the furniture for a room, as shown in \ref{fig2}. In order to synthesize a scene from an empty room according to the owner's suggestion, the assistant model should complete the following three steps. Firstly, partial scenes of the rooms are completed. Secondly, a high-level representation of the layout of a room in the form of a relation graph is extracted. Thirdly, the house owner's requirements for the indoor decoration are encoded and adapted in the model. As a consequence, the proposed system generates a relation graph following the owner's requirements and then produces a solution to the house layout.
\begin{figure}
\centering
\includegraphics[height=5.0cm]{fig3.jpg}
\caption{Graph extraction from the proposed rule of encoder hierarchy. The walls are at the first level, the doors are at the second level, the windows are at the third level and the objects are at the last level. }
\label{fig3}
\end{figure}
\begin{figure}
\centering
\includegraphics[height=5.0cm]{fig4.jpg}
\caption{Scene instantiation hierarchy from an abstract graph.}
\label{fig4}
\end{figure}

This assistant model starts with the extraction of the relation graphs from 3D indoor scenes. This extraction applies the geometric rules and a hierarchy rule to build a structural graph of the indoor scenes, as shown in Figure \ref{fig3} and Figure \ref{fig4}. Next, this corpus of the extracted graphs of the indoor scenes and the customers' requirements on the scenes are encoded. The encoded representation is applied to learn a generative model of graphs, as shown in Figure \ref{fig5}. The generated graph is therefore corresponding to the representation of an indoor scene given a particular condition. Finally, the abstract conditional graph and the representation of a suggestion are encoded to instantiate into a concrete scene, as shown in Figure \ref{fig6}.

\begin{figure}
\centering
\includegraphics[height=7.0cm]{fig5.jpg}
\caption{Conditional graph generation module.}
\label{fig5}
\end{figure}

Formally, given a set of indoor scenes ${(x_{1},y_{1}, s_{1}),\dots,(x_{N},y_{N},s_{N})}$ where $N$ is the number of the scenes, and $x_{i}$ is an empty indoor scene with basic elements including walls, doors and windows. $y_{i}$ is the corresponding layout of the furniture for $x_{i}$. Each $y_{i}$ contains the elements ${p_{j},s_{j},d_{j},l_{j}}$: $p_{j}$ is the position of the $jth$ element; $s_{j}$ is the size of the $jth$ element; $d_{j}$ is the direction of $jth$ element; and $l_{j}$ is the category label of the $jth$ element. Each element represents a furniture in an indoor scene $i$. Besides, $s_{i}$ is the requirement from the property owner that determines the layout type of the room. Representation of the each indoor scene ${(x_{i},y_{i},s_{i})}$ is first extracted to structural representation ${gp_{i},en_{i}}$. $gp_{i}$ is the graph representation of $y_{i}$, and $en_{i}$ is the code of $s_{i}$. A generative model is then trained to produce the abstract graph $gpout_{i}$ by applying hidden vectors $z$ and $s_{i}$. Finally, gievn $gpout_{i}$ and $s_{i}$, an instantiation model is learned to convert the abstract representation $gpout_{i}$ and $s_{i}$ to the completed scene $y_{i}$. An industrial rendering software is then applied to produce an industrial solution for the  property owner. The whole assistant system is aimed to help interior designers to produce the layout of the furniture in a shorter time at the industrial end.

\section{Graph Representation of Scenes}
In this section, we define the graph representation of the indoor scenes, and introduce a procedure of automatically extracting the graphs from 3D indoor scenes.

\subsection{Dataset}
We propose a dataset which is a collection of over fifteen thousand indoor-scenes designs from professional interior designers who use an industrial design software in a daily basis. These designs contain three common types of rooms including tatami, kitchen and balcony. Pre-processing on this data is performed to filter out uncompleted layout of the rooms and the mislabeled room types, resulting in 5000 kitchens (with 41 unique object categories), 5000 balconies (with 37 unique categories) and 1000 tatami rooms (with 53 unique categories). Each room has a list of properties including its category, the geometry of each object and the requirements for the layout. There are no hierarchical and semantic relationships among objects in each room.

\subsection{Graph Definition}
The hierarchical and semantic relationship among objects in a room is encoded in the form of a relation graph. In each graph, each node denotes an object, each edge denotes spatial or semantic relationship between objects. Each node is labeled with a category label of the object. In particular, the walls, doors, windows and different objects have different category labels. The distance between objects contains three labels including near, middle and further. The spatial relationship between objects is represented with this three labels. Similarly, the semantic relationship is represented with three labels as following, walls-doors, walls-windows, walls-objects and objects-objects.

\begin{figure}
\centering
\includegraphics[height=7.0cm]{fig6.jpg}
\caption{Conditional Scene Instantiation Module.}
\label{fig6}
\end{figure}

\subsection{Graph Extraction}
To convert an indoor scene into the above graph representation, We first label each node: the node of walls is $1$, the node of doors is $2$, the node of windows is $3$,  and the nodes of objects of different categories are  $4,5\dots,n-1,n$. The number of edge types is set to $9$ following the above definition, and there are $3$ types of spatial relationship and $3$ types of semantic relationship. For example, the edge type is $2$ if the nodes are a wall and a door, and the distance between them is middle. The edge type is $6$ if the nodes are a wall and an object and the distance between them is further.

The graph extracted is dense (average of $5$ edges node), and it contains many non-semantically meaningful relationships. This density may lead to problems for the learning of a generative graph model. The density is very likely to confuse a neural network as it is hard to recognize the most important structural relationships among many edges. Therefore, less important edges are pruned from the extracted graph. We set a pruning procedure for the extracted graph. Edges between nodes in which both nodes are the wall are deleted. Edges between nodes where the spatial relationship is further deleted. For each node that is labeled as a door or window, we keep the links to the closest wall and the object. Similarly, for each node that is labeled as objects, we keep the links to the closest objects and door/window. Finally, to ensure the graph connectivity, we make use of the same technique to reconnected the nodes as in \cite{10.1145/3306346.3322941}.

\section{Deep Graph Generation with Conditions}
Once the graph is extracted, the indoor scenes and the corresponding customers' requirements ${(x_{1},y_{1}, s_{1}),\dots,(x_{N},y_{N},s_{N})}$ are converted into graph representation and nodes ${(gp1_{1},gp2_{1}, en_{1}),\dots,(gp1_{N},gp2_{N},en_{N})}$. We focus on the problem of conditional structure generation of the completed indoor scene $y$.

Given a set of graph $gp2_{1},\dots,gp2_{n}$, where $gp2_{i}$ is the extracted graph representation of $y_{i}$ and $gp2_{i}=\{V_{j},E_{j}\}$ corresponds to the graph structure described by the set of nodes $V_{j}$ and the set of edges $E_{j}$. The semantic attribute of the graph is denoted as the code $en_{i}$. As already introduced above, $en_{i}$ describes the requirements from the corresponding property owner. It is basically the context of the graph.

A model $M$ is trained on a set of graphs with conditions $\{G_{i},E_{i},i=1,...,n\}$, and this $M$ is used to generate more graphs $g$ mimicking the structure of those in the training set. We apply a conditional graph generation model, which applies three tricks to generates a graph for different conditions. Firstly, latent space conjugation is applied to effectively convert node-level encoding into permutation-invariant graph-level encoding. This conjugation allows the learning on arbitrary numbers of graphs and generation of graphs with variable sizes. Formally, given a graph $G=\{V,E\}$, $G$ is regarded as a plain network with the adjacency matrix $A$. It generates node features $X=X(A)$ as the standard $k-$-dim spectral embedding based on $A$. The stochastic latent variable $Z$ is then introduced and $q(Z|X,A)=\prod_{i=1}^n q(z_{i}|X,A),z_{i} \in Z$ is the regarded as the node embedding of $v_{i} \in V$. A single distribution $\bar{z}$ is used to model all $z_{i}$'s by enforcing:
\begin{equation}
    q(z_{i}|X,A) \sim \mathcal{N}(\bar{z}|\bar{\mu},diag(\bar{\sigma}^{2}))
\end{equation}
where $\bar{\mu}=\frac{1}{n} \sum_{i=1}^{n} g_{\mu} (X,A)$, and $\bar{\mu}^{2}=\frac{1}{n_{2}} g_{\theta}(X,A)_{i}^{2}$. $g(X,A)=\bar{A}ReLU(\bar{A}XW_{0})W_{1}$ is a two-layer GCN model. $g_{\mu}(X,A)$ and $g_{\theta}(X,A)$ compute the matrices of mean and standard deviation vectors, which share the first-layer parameters $W_{0}$. $g(X,A)_{i}$ is the $i$-th row of $g(X,A)$. $\bar{A}=D^{-\frac{1}{2}}AD^{-\frac{1}{2}}$ is the symmetrically normalized adjacency matrix of $G$, where $D$ is its degree matrix with $D_{ii}=\sum_{j=1}^{n}A_{ij}$.

After sampling a desirable number of $z_{i}$'s to improve the capability of the graph decoder, a few fully connected neural network (FNN) layers $f$ are applied to $z_{i}$ before computing the logistic sigmoid function for link prediction which can be described by
\begin{equation}
    p(A|Z) = \prod_{i=1}^n \prod_{j=1}^n p(A_{ij}|z_{i},z_{j})
\end{equation}
where $p(A_{ij}=1 |z_{i},z_{j}) = \theta(f(z_{i})^{T}f(z_{j}))$, and $\theta(z)=\frac{1}{1+e_{-z}}$. The model is optimized by minimizing the minus variational lower bound:
\begin{equation}
    L_{vae} = L_{rec} + L_{prior} = \mathcal{E}_{q(Z|X,A)}[\log p(A|Z)] - D_{KL}(q(Z|X,A)||p(Z))
\end{equation}
where $L_{rec}$ is a link reconstruction loss and $L_{prior}$ is a prior loss based on the Kullback-Leibler divergence towards the Gaussian prior $p(Z)_{i=1}^{n}=\mathcal{N}(Z|0,I)^{n}$. The model now consists of a GCN-based graph encoder $\xi(A) = \frac{1}{n} \sum_{i=1}{n}g(X(A),A)_{i}$, and an FNN-based graph decoder/generator $\Upsilon(Z) = f(z_{i})^{T}f(z_{j})$.

GCN is leveraged by devising a permutation-invariant graph discriminator. This discriminator is applied to enforce the intrinsic structural similarity between $A^{'}$ and $A$ under arbitrary node ordering. In particular, a discriminator $D$ of a two-layer GCN is constructed followed by a two-layer FNN. This discriminator is trained together with the above encoder $\xi$ and generator $\Upsilon$ through following the GAN loss of a two-player minimax game:
\begin{equation}
    L_{GAN} = \log(D(A)) + \log(1-D(A^{'}))
\end{equation}
where $D(A) = f^{'}(g^{'}(X(A),A))$, and $X$, $g^{'}$ and $f^{'}$ are the spectral embedding for GCN and FNN, respectively.

\section{Scene Instantiation}
In this section, we describe the procedure of taking a relationship graph and instantiating it into an actual indoor scene. In the extraction of graph, the edge pruning steps is likely to lead to the incomplete graph. Besides, the graph representation of the scene is abstract. It does not contain enough relationship edges to uniquely determine the spatial positions and orientations of all object nodes. The scene instantiation is also required to follow the preferences of the property owner.

Formally, a procedure is required for sampling from the following conditional probability distribution corresponding to the property owner's requirements $en$:

\begin{equation}
    p(S|\Upsilon(V,E,en)) = p(S) en_{i} \in en \ 1(en_{i} \in S) v \in V \ 1(v \in S) (u,v,r) \in E (S, u, v)
\end{equation}

where $S$ is a scene, $\Upsilon(V,E,en)$ is a graph with vertices $V$, edges $E$, and requirements $en$. $r$ is a predicate function indicating whether the relationship implied by the edge $(u,v,r)$ is satisfied in $S$. $en_{i}$ is a sample of the requirements $en$.

\subsection{Object Instantiation Order}
Before the prediction of the location of objects in the room, the order of the instantiation of objects should be set. The algorithm of determining the order in which to instantiate objects follows a similar logic in \cite{10.1145/3306346.3322941}. It applies the structure of the graph, along with statistics about typical sizes for nodes of different categories. For example, in the instantiation of the tatami room, the order is set as tatami, work-desk, and cabinet.

\subsection{Neural-guided Object Instantiation}
After the order of instantiation is determined and an object of category $c$ is selected to add to the scene, the completed layout prediction of this object is expected to be produced following the requirements of the property owner's suggestion $en$. These prediction includes the location $p$, orientation $s$ and physical dimensions $d$.

Ideally, a generator function $g(p|en,s|en,d|(c,en),S,\Upsilon(V,E,en))$ is expected to produce the outputs values with probability proportional to the true conditional scene category $p(\bar(S) = S \cup \{c,p,s,d,en\}|\Upsilon(V,E,en))$. As already explored in the previous work through iterative object insertion \cite{Dai_2018_CVPR}, the location, orientation and dimensions are sampled iteratively according on the corresponding to $en$.

\subsubsection{Structural representation of requirements}
The requirements from the property owner are encoded as a vector in the graph generation phase. For example, if the balcony is in two categories leisure balcony and living balcony. The requirements vector is then one-hot vector $(1,0)$ and $(0,1)$, respectively. However, this one-hot vector loses much information in the conditional scene instantiation phase. Therefore, we develop a mixture embedding module which applies the embedding of global graph representation, local graph representation, and edge type representation.

We first employ a GCN \cite{8953909} network to produce the embedding of the global representation of the generated graph $gp$. Then, we employ the GCN \cite{8953909} and the setting instantiation order to produce the embedding of the local representation of the generated graph $gn$. Specifically, we apply the order to get the vector of the adjacency matrix of the selected objects. The embedding of the local graph representation can be obtained. The embedding of the edge-type is obtained in a similar method as in the previous model \cite{10.1145/3306346.3322941}. Note that this embedding follows a different rule as introduced above. Finally, we apply two-level FCN to combine these three types of embedding.

\subsubsection{Conditional instantiation}
After the structural representation of the suggestion is obtained, we train a model that is based on the condition of the requirements. This model receieves the inputs of the generated graph $gp$ and code the requirements $en$, and then predicts the location $p$, the orientation $s$, and the size $d$ of the selected object. We train both the embedding module and instantiation module together. Here the instantiation module is similar to the instantiation network in the previous work \cite{10.1145/3306346.3322941}.

\section{Evaluation}
\begin{figure}
\centering
\includegraphics[height=6.0cm]{fig7.jpg}
\caption{Generated layout examples with personalized preferences from the property owners.}
\label{fig7}
\end{figure}
In this section, we show both qualitative and quantitative results that demonstrate the unity of the proposed assistant model to auto-layout of the indoor scenes with property owners' personalized requirements. The generated examples are shown in Figure \ref{fig7}.

\subsection{Generation Accuracy}
The accuracy of the graph generation for each type of rooms is defined by:
\begin{equation}
    ACC_{G} = \frac{1}{N_{c}} \frac{\sum_{k=1}^{n} \text{label}(g_{k})=i}{N_{i}}
\end{equation}
where $N_{i}$ is the number of generated graph for label $i$ and $N_{c}$ is the number of the category of the graph.

We measure the accuracy of graphs for three types of rooms. For the evaluation of tatami room, we collect four types which are most popular among property owners from our sold interior decoration solutions. These four types are sleeping-functional tatami, storage-functional tatami, tea-functional tatami, and working-functional tatami. Similarly, There are three types of balconies chosen: the leisure-functional balcony, the washing-functional balcony, and the storage-functional balcony. There are two types of kitchen: the classical-functional kitchen and the multi-functional kitchen.
\begin{table*}

		\caption{Accuracy for tatami rooms.}
		\begin{center}
			\begin{tabular}{ccccc}
				\hline
                Averaged accuracy &For sleep&For tea&For storage&For work\\
                \hline
                $0.81$&$0.82$&$0.83$&$0.79$&$0.78$\\
                \hline
			\end{tabular}
			\label{table1}

		\end{center}
	\end{table*}

\begin{table*}

		\caption{Accuracy for balcony rooms.}
		\begin{center}
			\begin{tabular}{cccc}
				\hline
                Averaged accuracy&For leisure&For wash&For storage\\
                \hline
                $0.84$&$0.83$&$0.85$&$0.84$\\
                \hline
			\end{tabular}
			\label{table2}

		\end{center}
	\end{table*}

\begin{table*}

		\caption{Accuracy for kitchen rooms.}
		\begin{center}
			\begin{tabular}{ccc}
				\hline
                Averaged accuracy & For classical&For mixture\\
                \hline
                $0.86$&$0.88$&$0.84$\\
                \hline
			\end{tabular}
			\label{table3}

		\end{center}
	\end{table*}

In Table \ref{table1}, Table \ref{table2} and Table \ref{table3}, we show the averaged accuracy for different types rooms with different functionalities. Note that we measure $10000$ generated interior scenes for all functional-type of rooms.

\subsection{Perceptual Study}
In addition to the quantitative evaluation of the labeled graph generation, a two-alternative forced-choice (2AFC) perceptual study is conducted to compare the images from generated scenes with the corresponding scenes from the sold industrial solutions. The generated scenes are rendered using industrial rendering software. We remark that the rendering process is different from the previous work \cite{10.1145/3306346.3322941} that only produces solid color for the objects. Participants are shown two top-view scene images side by side and required to pick out the more plausible one. For each comparison and each room type, $10$ professional interior designers were recruited as the participants. As shown in Table \ref{table4}, the generated scenes are similar to the scenes from the sold solutions.

\begin{table*}
		\caption{Percentage ($\pm$ standard error) of 2AFC perceptual study where the real sold solutions are judged as more plausible than the generated scenes.}
		\begin{center}
			\begin{tabular}{lccc}
				\hline
				Room&Ours&PlanIT&Grains\\
                Tatami&$57.12\pm3.48$&$71.93\pm6.42$&$69.42\pm5.87$\\
                Balcony&$58.21\pm2.95$&$75.23\pm2.74$&$67.51\pm1.53$\\
                Kitchen&$54.21\pm4.51$&$74.68\pm4.62$&$77.94\pm2.90$\\
			\end{tabular}
			\label{table4}
		\end{center}
	\end{table*}

Another two-alternative forced-choice (2AFC) perceptual study is conducted to compare images from the generated scenes with the corresponding generative scenes of the state-of-the-art models. $10$ professional interior designers were recruited as the participants in this perceptual study. As Table \ref{table4} shows, the generated scenes of the proposed system outperforms the state-of-the-art models such as PlanIT \cite{10.1145/3306346.3322941} and Grains \cite{10.1145/3303766}.  It is obvious that our model is able to generate indoor scenes for a variety of function-category rooms of a particular room type while the baseline models are not able to produce such layout. We show an example of the generated tatami room with four functionalities in Figure \ref{fig1}.

\section{Discussion}
In this paper, we present an assistive method for interior decoration. The assistive method consists of the extraction of the abstract graph, conditional graph generation, and the conditional scene instantiation. The proposed system supports the interior designers in the industrial process to produce the decoration solution more quickly. In particular, this system produces the auto-layout of objects which is plausible according to the choice of the professional designers. Besides, the proposed system is shown to meet the personalized preferences from the property owners.

There are many challenges in automatic interior decoration. The proposed method is only able to support the auto-layout of objects in the interior scenes where the only global auto-layout follows the preferences of the property owners. The type, color, and brand of each object for the personalized preferences are not supported. In addition, the proposed method produces good results when the shape of the room is towards regular while there are a variety of different shapes in the real world. It is worthwhile to explore smarter systems to solve these challenges.

\bibliographystyle{splncs04}


\section{Introduction}

Generalization in neural networks trained with stochastic gradient descent (SGD) is not well-understood. For example, the generalization gap, i.e., the difference between training and test error depends critically on the dataset and we do not understand how. This is most clearly seen when we fix all aspects of training (e.g. architecture, optimizer, learning rate schedule, etc.) and vary only the dataset. In a typical experiment designed to test this, training on a real data set (e.g., ImageNet) leads to a relatively small generalization gap, whereas training on randomized data (e.g., ImageNet with random labels) leads to a much larger gap~\citep{Zhang17, Arpit17}.

The mystery is that in both cases (real labels and random) the training accuracy is close to 100\% which implies that the network and the learning algorithm have sufficient effective capacity~\citep{Arpit17} to memorize the training sets, i.e., to fit an arbitrary mapping from the input images to labels. But, what then, is the mechanism that from among all the maps consistent with the training set, allows SGD to find one that generalizes well (when such a well-generalizing map exists)?

This question has motivated a lot of work (see e.g. ~\citet{Zhang17,Arpit17,Bartlett17,Kawaguchi17,Neyshabur18,Arora18,Belkin19,Rahaman19}) but no satisfactory answer has emerged. As \citet{Nagarajan19} point out, traditional approaches based on uniform convergence may not suffice, and new ideas are needed. A promising line of attack is via algorithmic stability~\cite{Bousquet02}, but traditional stability analysis of SGD (e.g., \citet{Hardt16, Kuzborskij18}) does not account for the dataset, and without that, one cannot hope to get more than a vacuous bound.

Recently, a new approach called Coherent Gradients (CG) has been proposed that takes into account the training dataset in reasoning about stability~\citep{Chatterjee20, Zielinski20}. By analogy to Random Forests which also show dataset  dependent generalization, CG posits that neural networks try to extract commonality from the dataset during the training process.

The key insight is that, since the overall gradient for a single step of SGD is the sum of the per-example gradients, it is strongest in directions that reduce the loss on multiple examples if such directions exist. Intuitively, at one extreme, if all the per-example gradients are aligned we get perfect  stability (since dropping an example doesn’t affect the overall gradient) and thus perfect generalization. At the other extreme, if all the per-example gradients are pairwise orthogonal, we get no stability (since dropping an example eliminates any descent down its gradient), and thus pure memorization.

Thus CG provides a simple, unified explanation for both memorization and generalization. However, at the same time, CG leads to some basic empirical questions:

\begin{enumerate}

\item  {\em What does the alignment of per-example gradients, i.e., {\em coherence} look like in practice?}

As was noted in~\citet{Chatterjee20}, we expect a real dataset to have more coherence than a dataset with random labels, but how big is this difference quantitatively? Is coherence in the random label case like that in the pairwise orthogonal case described above? How does it vary with layer or architecture?

\item {\em Is the coherence constant throughout training, or does it vary? If so, how?}

The key insight of CG (as described above) is a point-in-time observation, but in order to get a full picture of generalization we need to analyse the entire training trajectory. For example, one might imagine that as more and more training examples are fitted, coherence decreases, but is it possible for it to increase in the course of training?

\end{enumerate}

In this paper, we propose a new metric called $\mcoherence$ to experimentally study gradient coherence. The metric admits a very natural intuitive interpretation that allows us to gain insight into the questions above.
While we confirm our intuitions in many cases, we also find some surprises.
These observations help us formulate more precisely what is missing from the CG explanation for generalization, and thus point the way to future work in this direction.

\section{Prior Work on Metrics for Experimentally Measuring Coherence}
\label{sec:prev}

{\bf Pairwise Dot Product.} An obvious starting point to quantify the alignment or coherence of a set of gradients is their average pairwise dot product. Since this has a nice connection to the loss function, we start by reviewing the connection, and also set up notation in the process.

Formally, let $\mathcal{D}(z)$ denote the distribution\footnote{We would like to quantify gradient coherence for both populations and samples. Therefore, $\mathcal{D}$ can either be a population distribution (typically unknown) or a sample  (i.e., empirical) distribution.} of examples from a finite\footnote{We assume finiteness for simplicity  since it does not affect generality for practical applications.} set $Z$, and assume without loss of generality that ${\rm support}(\mathcal{D}) = Z$.
For a network with $d$ trainable parameters,
let $\ell_z(w)$ be the loss for an example $z \sim \mathcal{D}$ for a parameter vector $w \in \mathbb{R}^d$. For the learning problem, we are interested in minimizing the expected loss $\ell(w) := \E_{z \sim \mathcal{D}}[\ell_z(w)]$.
Let $g_z := [\nabla \ell_z](w)$ denote the gradient of the loss on example $z$, and $g := [\nabla \ell](w)$ denote the overall gradient. From linearity, we have,
\[
g = \E_{z \sim \mathcal{D}}\ [\ g_z\ ]
\]
Now, suppose we take a small descent step $h = - \eta g$ (where $\eta > 0$ is the learning rate). From the Taylor expansion of $\ell$ around $w$, we have,
\begin{equation}
\ell(w + h) - \ell(w) \approx g \cdot h
= - \eta\  g \cdot g
= - \eta \E_{z \sim \mathcal{D}}\ [\ g_z\ ] \cdot \E_{z \sim \mathcal{D}}\ [\ g_z\ ]
= - \eta \E_{z \sim \mathcal{D}, z' \sim \mathcal{D}}\ [g_z \cdot g_{z'}]
\label{eq:dloss}
\end{equation}
where the last equality can be checked with a direct computation.
Thus, the following are approximately equivalent:
\begin{itemize}
    \item reduction in loss (due to a small step) divided by the learning rate,
    \item squared $\ell^2$ norm of the expected gradient, and,
    \item expected pairwise dot product (where the expectation is over {\em all} pairs).
\end{itemize}

{\bf Example.}~\citep{Chatterjee20} Consider a sample with $m$ examples $z_i$ where $1 \leq i \leq m$. Let $g_i$ be the gradient of $z_i$ and further that
$\|g_i\| = \|u\|$ for some $u$.
If all the $g_i$ are the same, then $g \cdot g = \|u\|^2$. However, if they are pairwise orthogonal, i.e., $g_i \cdot g_j = 0$ for $i \neq j$, then $g \cdot g = \frac{1}{m} \|u\|^2$.
\qed

As this illustrates, the average expected dot product can vary significantly depending on the coherence. However, as a metric for coherence it is rather fragile. For example, just re-scaling the loss can drastically alter the value of the metric.
Therefore, it can only be used to reason about coherence in very limited settings.
For e.g., \citet{Chatterjee20, Zielinski20} use it to verify that adding increasing amounts of label noise to a dataset reduces coherence but in order to do so they keep everything else the same, and limit their considerations to the start of training.
But, to study the evolution of coherence, even over a single training run requires normalization since the magnitude of the gradients changes significantly in the course of training (e.g., see Appendix).

{\bf Stiffness.} \citeauthor{Fort19} in their preprint \citeyearpar{Fort19} study two variants of the average pairwise dot product that they call {\em sign stiffness} and {\em cosine stiffness}. In our notation these are
\small
\[
S_{\rm sign} :=
\E_{\substack{z \sim \mathcal{D}, z' \sim \mathcal{D}\\ z \neq z'}}[\ {\rm sign}(g_z \cdot g_{z'})\ ]
\ \
{\rm and}
\ \
S_{\rm cos} :=
\E_{\substack{z \sim \mathcal{D}, z' \sim \mathcal{D}\\ z \neq z'}}\left[ \ \frac{g_z}{\|g_z\|} \cdot \frac{g_{z'}}{\|g_{z'}\|}\ \right].
\]
\normalsize
These are meant to capture how a small gradient step based on one input example affects the loss on a {\em different} input example.
Although \citeauthor{Fort19} do not describe why they choose to transform the gradients in these specific ways, we expect it is to normalize the dot product so that it can be tracked in the course of training.
In their experience, they found sign stiffness to be more useful to analyze stiffness between classes whereas cosine stiffness was more useful within a class.

{\bf Gradient Confusion.} \citeauthor{Sankararaman19} in their preprint \citeyearpar{Sankararaman19} introduce the notion of a gradient confusion bound. The {\em gradient confusion bound} is $\zeta \geq 0$ if for all
$z, z' \in Z$ and $z \neq z'$, we have,
$g_z \cdot g_{z'} \geq -\zeta$.
They use this concept to study theoretically the convergence rate of gradient descent, but in their experimental results they measure the minimum cosine similarity between gradients, i.e.,
\small
\[
\min_{\substack{z \in Z, z' \in Z \\ z \neq z'}}
\left[ \ \frac{g_z}{\|g_z\|} \cdot \frac{g_{z'}}{\|g_{z'}\|}\ \right]
\]
\normalsize
We note that the non-linearities (and to a lesser extent the $z \neq z'$ restriction) make it hard to tie stiffness or minimum cosine similarity to what happens during training; specifically, to the change in the loss function as a result of a gradient step which is the expectation over {\em all} per-example gradients.

\section{A New Metric for Coherence}
\label{sec:new}

The key insight behind our proposal is that there is a natural scaling factor that can be used to normalize the expected dot product of per-example gradients (i.e., the quantity in (\ref{eq:dloss})) that preserves the connection to the loss.
Consider the Taylor expansion of each individual loss $\ell_z$ around $w$ when we take a small step $h_z$ down {\em its} gradient $g_z$:
\[
\ell_z(w + h_z) - \ell_z(w) \approx g_z \cdot h_z
= - \eta\  g_z \cdot g_z
\]
Taking expectations over $z$ we get,
\begin{equation}
\E_{z \sim \mathcal{D}} [ \ell_z(w + h_z) - \ell_z(w) ]
= - \eta\ \E_{z \sim \mathcal{D}} [ g_z \cdot g_z ]
\label{eq:dloss_ez}
\end{equation}
The quantity in (\ref{eq:dloss_ez}) has a simple interpretation: It is the reduction in the {\em overall} loss $\ell$ if each example $\ell_z$ could be optimized independently.
As might be expected intuitively, it is an upper bound on the quantity in  (\ref{eq:dloss}) and is tight when all the per-example gradients are identical. We prove this formally in \S\ref{sec:facts}.
Thus, it serves as a natural scaling factor for the expected dot product, and we obtain a normalized metric for coherence (denoted by $\alpha$) from (\ref{eq:dloss}) and (\ref{eq:dloss_ez}):
\small
\begin{equation}
\boxed{
\
\alpha :=
\frac{\ell(w + h) - \ell(w)}{\displaystyle \E_{z \sim \mathcal{D}}\ [ \ell_z(w + h_z) - \ell_z(w) ]} =
\frac{\displaystyle \E_{z \sim \mathcal{D}, z' \sim \mathcal{D}}\ [g_z \cdot g_{z'}]}
{\displaystyle \E_{z \sim \mathcal{D}}\ [ g_z \cdot g_z ]}
=
\frac{\displaystyle \E_{z \sim \mathcal{D}}\ [\ g_z\ ] \cdot \E_{z \sim \mathcal{D}}\ [\ g_z\ ]}
{\displaystyle \E_{z \sim \mathcal{D}}\ [ g_z \cdot g_z ]}
=
\frac
{\displaystyle \E_{z \sim \mathcal{D}}\ [\ g_z \cdot g_{\ } ]}
{\displaystyle \E_{z \sim \mathcal{D}}\ [\ g_z \cdot g_z ]}
\
}
\label{eq:metric}
\end{equation}
\normalsize

{\em Thus, $\alpha$ is the change in the overall loss due to a small gradient step as a fraction of the maximum possible change in loss if each component of the loss could be optimized independently}.

As noted before, $0 \leq \alpha \leq 1$, and the maximum is achieved when all the gradients are identical, and the minimum is achieved when the expected gradient is 0, i.e., a stationary point is reached.

{\bf A natural scale for $\alpha$.}
Once again, consider a sample with $m$ examples $z_i$ where $1 \leq i \leq m$. Let $g_i$ be the gradient of $z_i$. Suppose further that the $g_i$ are pairwise orthogonal i.e. $g_i \cdot g_j = 0$ for $i \neq j$.
It is easy to check that $\alpha = 1/m$. For a sample of size $m$, we call this value of $\alpha$ the {\em orthogonal limit}.

Since in the orthogonal case, each example is optimized independently, going down the expected gradient is $1/m$ times as slow as optimizing each independently. If the gradients are better aligned, we expect them to help each other resulting in an $\alpha$ greater than the orthogonal limit.

{\bf Example (Commonality).} For $1 \le i \le m$, suppose each $g_i$ has a common component $c$ and an idiosyncratic component $u_i$, i.e., $g_i = c + u_i$ with $u_i \cdot u_j = 0$ for $1 \le j \le m$ and $j \ne i$; $u_i \cdot c = 0$; and say, $u_i \cdot u_i = \|u\|^2$ for some $u$. It is easy to see that $\alpha$ in this case is
$ \frac{1}{m} \left[ 1 + (m - 1) \cdot f  \right] $
where $f = \|c\|^2 / (\|c\|^2 + \|u\|^2)$.
\qed

These examples along with the observation that $0 \le \alpha \le 1$ suggests a more evocative (even if less accurate and less general) interpretation: In a given sample, $\alpha$ is the average fraction of examples that each example helps or supports.
Thus, when analyzing experimental data, for a sample of size $m$, it is convenient to define a new quantity $\mcoherence$ as follows:
\small
\[
\boxed{
\qquad
\mcoherence := m \cdot \alpha
=
m \cdot \frac{\displaystyle \E_{z \sim \mathcal{D}, z' \sim \mathcal{D}}\ [g_z \cdot g_{z'}]}
{\displaystyle \E_{z \sim \mathcal{D}}\ [ g_z \cdot g_z ]}
\qquad
}
\]
\normalsize
Thus $m$-coherence in the orthogonal limit is 1 and in the identical case is $m$. {\em Intuitively, $m$-coherence of a sample is the number of examples (including itself) that any one example helps on average.}

{\bf Advantages.} $\alpha$ and $\mcoherence$ have several advantages over the metrics discussed in \S\ref{sec:prev}:

\begin{itemize

\item {\bf Computational Efficiency.} For a sample of size $m$, due to (\ref{eq:metric}), $\alpha$ can be computed exactly in $O(m)$ time in contrast to $O(m^2)$ time required for stiffness and cosine dot products. Furthermore, it can be computed in a streaming fashion by keeping two running sums, so the per-example gradients need not be stored. Thus, in our experiments we are able to use sample sizes a couple of orders of magnitude higher than those in~\citet{Fort19} and \citet{Sankararaman19}.

\item {\bf Mathematical Simplicity.} We believe our definition is cleaner mathematically. This allows us to reason about the metric more easily. For example,

\begin{enumerate}

    \item We can show that the coherence of minibatch gradients is greater than that of individual examples (Corollary~\ref{cor:amp}). Therefore, care must be taken
    if minibatch gradients are used in lieu of example gradients in computing coherence (e.g. as in \citet{Sankararaman19}).

    \item Explicitly ruling out $z \neq z'$ as in done in stiffness and cosine similarity to eliminate  self-correlation is unnatural and can get tricky in practice due to near-duplicates or multiple examples leading to same or very similar gradients.
    %
    We obtain meaningful values without imposing those conditions, but if one insists on removing self-correlations, then subtracting $1/m$ from $\alpha$ or 1 from $m$-coherence is a more principled way to do it.

    \item The non-linearities in stiffness and cosine similarity amplify small per-example gradients potentially overstating their importance, and lead to a discontinuity (or undefined behavior) with zero gradients. However, we can cleanly account for the effect of
    negligible gradients in our observations (e.g. see Lemma~\ref{lem:zero}).

\end{enumerate}

\item {\bf Interpretability.} Finally, as discussed in detail above, they are normalized and yet easily interpretable due to the natural connection with loss.

\end{itemize}

{\bf Prior Work on Gradient Diversity.} While writing this paper we discovered that the reciprocal of $\alpha$ appears in the theory literature as {\em gradient diversity}. This was used by~\cite{Yin18} in theoretical bounds to understand the effect of mini-batching on convergence of SGD. (A similar result appears for least squares regression in \citet{Jain18}.)
They show that the greater is the gradient diversity, the more effective are large mini-batches in speeding up SGD. Although they support their theoretical analysis with experiments on {\sc cifar}-10 (where they replicate $1/r$ of the dataset $r$ times and show that greater the value of $r$ less the effectiveness of mini-batching to speed up) they never actually measure the gradient diversity in their experiments (or further study its properties).
Also, note that for our purposes $\alpha$ is a better choice than $1/\alpha$ -- not just because coherence rather than incoherence is
what leads to generalization -- but also since the latter can diverge: $g$ can be 0 without all $g_z$ being zero (e.g. at the end of training in an under-parameterized setting).

\section{A More General Setting for Coherence and Some Basic Facts}
\label{sec:facts}

Our notion of coherence is not specific to gradients (or optimization) but extends naturally to vectors in Euclidean spaces.
Let $\mathcal{V}$ be a probability distribution on a collection of $m$ vectors in an Euclidean space.
In accordance with (\ref{eq:metric}), we define the {\em coherence} of $\mathcal{V}$ (denoted by  $\alpha(\mathcal{V})$) to be
\small
\begin{equation}
\alpha(\mathcal{V}) = \frac{\displaystyle \E_{v \sim \mathcal{V}, v' \sim \mathcal{V}} \  [ v \cdot v' ]}{\displaystyle \E_{v \sim \mathcal{V}} \  [ v \cdot v]}
\label{eq:definealpha}
\end{equation}
\normalsize
Note that $\E[v \cdot v] = 0$ implies $\E[v \cdot v'] = 0$.
In what follows, we ignore the technicality of the denominator being 0 by always assuming that there is at least one non-zero vector in the support of $\mathcal{V}$ (which also held in our experiments).
We list some basic facts.

\vspace{5mm}
\begin{theorem}[Boundedness]
We have $0 \leq \alpha(\mathcal{V}) \leq 1$. In particular, $\alpha(\mathcal{V}) = 0$ iff $\E_{v \sim \mathcal{V}} [v] = 0$ and $\alpha(\mathcal{V}) = 1$ iff all the vectors are equal.
\end{theorem}

\begin{proof}
Since $v \cdot v \geq 0$ for any $v$, we have $\E_{v \sim \mathcal{V}}[v \cdot v] \geq 0$. Furthermore, it is easy to verify by expanding the expectations (in terms of the vectors and their corresponding probabilities) that
\begin{equation}
\displaystyle \E_{v \sim \mathcal{V}, v' \sim \mathcal{V}} \  [ v \cdot v' ] = \E_{v \sim \mathcal{V}} [v]\ \cdot \E_{v \sim \mathcal{V}} [v]
\geq 0.
\label{eq:norm}
\end{equation}
Therefore, $\alpha(\mathcal{V}) \geq 0$.
Likewise, another direct computation shows that
\begin{equation}
0 \leq
\E_{v' \sim \mathcal{V}}\ \left[
  (\E_{v \sim \mathcal{V}}[v] - v')
        \cdot
  (\E_{v \sim \mathcal{V}}[v] - v')
\right]
=
\E_{v \sim \mathcal{V}}[v \cdot v] -
\E_{v \sim \mathcal{V}}[v]\ \cdot \E_{v \sim \mathcal{V}}[v]
\end{equation}
Since from Equation~\ref{eq:norm} we have $\E[v] \cdot \E[v] = \E[v \cdot v']$, it follows that $\alpha(\mathcal{V}) \leq 1$.
Furthermore, since each term of the expectation on the left is
non-negative, equality is attained only when all the vectors are equal.
\end{proof}

\vspace{5mm}
\begin{lemma}[Scale Invariance]
For non-zero $k \in \mathbb{R}$,
let $k \mathcal{V}$ denote the distribution of the random variable $k v$ where $v$ is drawn from $\mathcal{V}$.
We have $\alpha(k \mathcal{V}) = \alpha(\mathcal{V})$.
\label{lem:scale}
\end{lemma}

\begin{proof}
\begin{equation}
\alpha(k\mathcal{V})
= \frac{\displaystyle \E_{v \sim k \mathcal{V}, v' \sim k \mathcal{V}} \  [ v \cdot v' ]}{\displaystyle \E_{v \sim k \mathcal{V}} \  [ v \cdot v]}
= \frac{\displaystyle \E_{v \sim \mathcal{V}, v' \sim \mathcal{V}} \  [ k v \cdot k v' ]}{\displaystyle \E_{v \sim \mathcal{V}} \  [ k v \cdot k v]}
= \frac{\displaystyle \E_{v \sim \mathcal{V}, v' \sim \mathcal{V}} \  [ v \cdot v' ]}{\displaystyle \E_{v \sim \mathcal{V}} \  [ v \cdot v]}
= \alpha(\mathcal{V})
\end{equation}
\end{proof}

\vspace{5mm}
\begin{theorem}[Stylized mini-batching]
Let $v_1, v_2, .., v_k$ be $k$ i.i.d. variables drawn from $\mathcal{V}$. Let $\mathcal{W}$ denote the distribution of the random variable $w = \frac{1}{k} \sum_{i=1}^{k} v_i$. We have,
\small
\begin{equation}
    \alpha(\mathcal{W}) =
    \alpha(k \mathcal{W}) =
    \frac{k \cdot \alpha(\mathcal{V})}{1 + (k - 1) \cdot \alpha(\mathcal{V})}
\end{equation}
\normalsize
\end{theorem}

\begin{proof}
The first equality follows from Lemma~\ref{lem:scale}. For the second equality, we have,
\begin{equation*}
\alpha(k \mathcal{W}) =
\frac{\displaystyle \E_{\substack{w \sim k \mathcal{W}, \\ w' \sim k \mathcal{W}}} \  [ w \cdot w' ]}{\displaystyle \E_{w \sim k \mathcal{W}} \  [ w \cdot w]} =
\frac{\displaystyle \E_{\substack{v_1, .., v_k, \\ v_1', .., v_k'}} \  [ (\sum_i v_i) \cdot (\sum_i v_i') ]}{\displaystyle \E_{v_1, .., v_k} \  [(\sum_i v_i) \cdot (\sum_i v_i)]} =
\frac{\displaystyle k^2 \E_{\substack{v \sim \mathcal{V},\\v' \sim \mathcal{V}}} \  [ v \cdot v' ]}
{\displaystyle k \E_{v \sim \mathcal{V}} \  [ v \cdot v] + k\cdot(k - 1)\E_{\substack{v \sim \mathcal{V},\\v' \sim \mathcal{V}}} \  [ v \cdot v' ]}
\end{equation*}
By dividing the numerator and denominator of the last expression by $\displaystyle k \E_{v \sim \mathcal{V}} \ [v \cdot v]$ the required result follows.
\end{proof}

\vspace{5mm}
\begin{corollary}[Minibatch amplification]
\label{cor:amp}
$\alpha(\mathcal{W}) \geq \alpha(\mathcal{V})$ with equality iff $\alpha(\mathcal{V}) = 0$ or $\alpha(\mathcal{V}) = 1$.
\end{corollary}

\begin{proof}
From the previous theorem, the transformation in coherence due to stylized mini-batching is given by the map $\alpha \mapsto \frac{k \cdot \alpha}{1 + (k - 1) \cdot \alpha}$.
Now, since $\alpha \leq 1$, we have $k \geq 1 + (k - 1) \cdot \alpha$, and since $\alpha \geq 0$, multiplying both sides by $\frac{\alpha}{1 + (k - 1) \cdot \alpha}$ we have $\frac{k \cdot \alpha}{1 + (k - 1) \cdot \alpha} \geq \alpha$. Finally, it is easy to check that the only two fixed points of the map are $\alpha = 0$ and $\alpha = 1$.
\end{proof}

{\bf Remark.} This formulation provides a nice perspective on the type of results proved in \citet{Yin18} and \citet{Jain18}. When
$\alpha \ll 1/k$ but non-zero (i.e., we have high gradient diversity), creating mini-batches of size $k$ increases coherence almost $k$ times. But, when  $\alpha \approx 1$ (i.e., low diversity) there is not much point in creating mini-batches since there is little room for improvement.

\vspace{5mm}
\begin{lemma}[Effect of zero gradients]
\label{lem:zero}
If $\mathcal{W}$ denotes the distribution where with probability $p > 0$ we pick a vector from $\mathcal{V}$ and with probability $1 - p$ we pick the zero vector then $\alpha(\mathcal{W}) =  p \cdot \alpha(\mathcal{V})$.
\end{lemma}

\begin{proof}
\begin{equation}
\alpha(\mathcal{W})
= \frac{\displaystyle \E_{w \sim \mathcal{W}, w' \sim \mathcal{W}} \  [ w \cdot w' ]}{\displaystyle \E_{w \sim \mathcal{W}} \  [ w \cdot w]}
= \frac{\displaystyle p^2 \cdot \E_{v \sim \mathcal{V}, v' \sim \mathcal{V}} \  [ v \cdot v' ]}{\displaystyle p \cdot \E_{v \sim \mathcal{V}} \  [ v \cdot v]}
= p \cdot \alpha(\mathcal{V})
\end{equation}
\end{proof}

{\bf Example (Coherence Reduction).} If we add $k$ zero gradients to the collection of gradients constructed in
the example of \S\ref{sec:new} (Commonality), using Lemma~\ref{lem:zero}, we get,
\small
\[
\alpha = \frac{m}{m + k} \cdot \frac{1}{m} \left[ 1 + (m - 1) \cdot f  \right]
= \frac{1}{n} \left[ 1 + (n - k - 1) \cdot f  \right]
\]
\normalsize
where $n = m + k$ is the size of this new sample. For a fixed $n$, as $k$ increases, $\alpha$ decreases going down to $1/n$ (the orthogonal limit) when all but one vector in the sample is zero,
i.e., $k = n - 1$.

\section{Experimental Results}

\begin{figure}[t]
\centering
\includegraphics[width=1\textwidth]{plots/v67/figure1_v67.pdf}
\vskip -0.1in
\caption{The evolution of alignment of per-example gradients of a ResNet-18 during training as measured with $\mcoherence$ on samples of size $m=40,356$ on 3 variants of ImageNet with different amounts of label noise. Our main finding is that coherence not only decreases in the course of training (as might be expected when examples get fit), but it also increases. The peak is reached rapidly with real labels (within the first 100 steps) and slowly with random labels (over many epochs). Horizontal lines for $m$-coherence are shown at 1 (the orthogonal limit) and at $m$. Vertical lines indicate sharp reductions in learning rate. Light dots show the results of 4 other runs to understand sensitivity w.r.t. randomness in initialization and mini-batch construction.}
\label{fig:resnet_full}
\vspace*{-0.2in}
\end{figure}

We now use $\mcoherence$ to experimentally study the evolution of coherence.

{\bf Methodology.} We train ResNet-18 models on ImageNet with original labels (0\% noise), and
two derived datasets: one with half the training labels randomized (50\% noise), and another with all the training labels randomized (100\% noise).\footnote{We use the original ImageNet validation set as our test set in all cases.} We using SGD with momentum (0.9), a batch size of 4096, and the learning rate schedule proposed in~\cite{Goyal17}.
We turn off augmentation and weight decay to observe memorization in the noisy cases within a reasonable number of steps.
For each dataset, we track $\mcoherence$ on a random (but fixed) set of $m = 40,356$ {\em training} examples.

Figure~\ref{fig:resnet_full} shows the data from our experiments. Each column corresponds to a different experiment and the rows show loss, accuracy, and $\mcoherence$ for the entire model and some specific layers.

{\bf Real Labels.}
Our first experiment (shown in the {\em second} column of Figure~\ref{fig:resnet_full} for reasons that will become clear shortly) measures the $\mcoherence$ (row 3) for training with 0\% noise, i.e., the real ImageNet labels.
The initial coherence in epoch 1 is very high, almost $10^4$ and it decreases as more training examples get fit.
We note that although there is some fluctuation in the coherence, it stays high (above $10^2$ and often above $10^3$) until well after the accuracy crosses the 50\% mark.
It settles at 1 after all the examples are fit.

The high initial coherence agrees well with the intuition from CG that real datasets have good per-example gradient alignment since that is what is necessary for good generalization as per the theory.
The subsequent decrease in coherence in the course of training is expected from Lemma~\ref{lem:zero} under the assumption that the gradients of fitted examples become small.

{\bf Random Labels.}
Our second experiment (column 6) shows that with random labels, the initial coherence in epoch 1 is low (between 1 and 10). It increases steadily until it reaches a peak in epochs 40 to 60 (between $10^2$ and $10^3$) followed by a decrease.

The low initial coherence (near the orthogonal limit) agrees well with CG as discussed in the introduction but the subsequent increase is surprising (though not in contradiction with CG as discussed later). The increase is not small since at its peak each example is helping hundreds of other examples (though it is well below the peak seen with real labels).
Once again, as examples get fitted, coherence decreases as expected from Lemma~\ref{lem:zero}, though not back down to 1, likely since our training only goes on till about 80\% accuracy is reached.

The increase in one case and not the other leads to a natural question with implications about the dynamics of SGD:
{\em Is the evolution of coherence fundamentally different between the well-generalizing case (real labels) and the memorization case (random labels)?}

{\bf Early Training.} To study this question, we took a closer look at the 1st epoch. We recorded $\mcoherence$ at initialization (i.e., before the first step) and, thereafter, for every step in the epoch. Since this requires computing the per-example gradients for $\approx$ 40K examples after every step, this was our most computationally expensive experiment taking 2-3 days per run (using TPUs).
The results are shown in columns 1 (real labels) and 5 (random).

\begin{wrapfigure}{R}{0.65\textwidth}
\centering
\includegraphics[width=0.5\textwidth]{plots/v67/inception_vs_resnet_v67.pdf}
\caption{The early trajectory of an Inception model shows a similar (post-transient) increase as ResNet. The overall trajectories are also similar, and it is interesting to note that we
get similar values for coherence although the two architectures are very different.}
\label{fig:inception}
\vspace*{-0.1in}
\end{wrapfigure}

In the first 25 steps, in {\em both} cases, we find that coherence shows a steep fall from nearly $m$ down to less than 10. This is accompanied by a sharp decrease in training loss (cross-entropy) (row 1) from around 16 to ${\rm ln}(1000) \approx 6.9$ (the value expected from a uniform distribution at the outputs).

However, after this initial transient, we find that in both cases the coherence starts from the low point and starts rising. The rise is much faster for real labels than for random labels. We ran additional experiments with 50\% noise (column 3) and with 25\% and 75\% noise (see Appendix) to confirm that the slope with which coherence increases depends inversely on the amount of label noise.

We believe the initial transient is likely due to all outputs (including the expected classes) being assigned a probability close
to 0 by the network at initialization (which would be consistent with the loss being above that of the uniform distribution). However, the reasons for the subsequent increase in coherence are not clear. We discuss in more detail later.

{\bf The Overall Evolution.} If we combine the data from early training (after the initial transient) with the rest of training (e.g., in row 3 we jointly view columns 1 and 2), we find a remarkably consistent pattern across all noise levels: $\mcoherence$ follows a broad parabolic trajectory (albeit with some local variation and noise) where it starts small (around 1), increases to a maximum, and then decreases back to 1.
Thus, there is always an initial increase in coherence, just on different timescales.

{\em From this point of view, the evolution of coherence in the memorization case does not look
fundamentally different from that in the well-generalizing case.}

{\bf Impact of Layers.} The bottom 3 rows of Figure~\ref{fig:resnet_full} shows coherence by layer for 3 illustrative layers (the first convolution layer, a convolution layer in the middle, and the final fully connected layer).
Although the specific values are different across the layers,\footnote{Since,
the interpretability of $\mcoherence$ allows for meaningful comparisons between layers, we
can get some additional insight into the dynamics of training by studying these values. We do so in the Appendix.}
we notice that the broad trajectory observed for the coherence of the entire model holds for each individual layer. Thus, the trajectory (and in particular the increase) is not driven by one specific layer.

{\bf Impact of Architecture.} We studied one other architecture, Inception, and found a similar increase in coherence and indeed a similar broad trajectory (Figure~\ref{fig:inception}). We note here that Inception does not show the strong initial transient seen in ResNet.

{\bf Coherence on Test Set.} For completeness, we also measured the coherence on $m$ examples not used for training (drawn from the ImageNet validation set). They are shown in columns 1 and 2 of Figure~\ref{fig:resnet_full} as ``test.'' We defer the discussion to the Appendix.

{\bf Reconciliation with Other Studies.} Finally, we note that it is difficult to compare our experimental results with \citet{Fort19} and \citet{Sankararaman19}, since we use different metrics, sample sizes, datasets (ImageNet v/s {\sc cifar}), and study different effects. But, in as much as they can be compared, we did not find contradictions. Please see the Appendix for more details.

\section{Discussion and Future Work}

{\bf Coherence, Generalization and CG.}
At a high-level, our experiments provide additional evidence for the connection between the alignment of per-example gradients and generalization~\citep{Fort19, Chatterjee20}. But as our data shows this connection is complicated.

According to CG, the generalizability of SGD is an ``inductive invariant'' that
the transition dynamics attempts to maintain at each step (as far as possible, given the coherence at that time).
Thus, if early on in training, there is low coherence (causing the inductive invariant to be violated) then all bets about generalization are off even if there is relatively high coherence later on.
From this perspective, the high coherence observed in the random label case does not contradict CG.

At the same time, the low coherence seen on real data after the initial transient may be viewed as contradicting the theory. However, we conjecture that since it only persists for a relatively few steps (about 100 steps compared to many epochs for random) there is not enough time for overfitting to occur. Similarly, the extended period of low coherence after the data has been fit does not totally destroy generalization since by that time the gradients are small.

Therefore, it is interesting to consider metrics to predict generalization that are derived from coherence but account for the distance travelled in parameter space.
Finally, an important test of CG comes from causal interventions to suppress ``weak'' gradient directions (directions supported by a few examples) which are shown to prevent overfitting (e.g. winsorized gradients~\citep{Chatterjee20} and RM3~\citep{Zielinski20}).
It would be interesting to study these through the lens of $\mcoherence$

{\bf Evolution of Coherence.}
The evolution of coherence appears to be controlled by two opposing forces.
On the one hand, as training progresses and examples get fit, coherence is consumed (as per Lemma~\ref{lem:zero} and the Coherence Reduction example).
On the other hand, as our experiments show, coherence is
also created during training.

One may imagine an uneasy equilibrium between these opposing tendencies leading to expansion and contraction in coherence. As soon as significant coherence builds up, it leads to an increase in the effective learning rate (higher relative gradient norm) leading to faster consumption.
Ultimately, of course, consumption wins out since a stable state is only reached when the gradient becomes small, falling to the orthogonal limit (if the system is sufficiently over-parameterized and we are in the interpolation regime) or below (if under-parameterized and improving loss on one example can only come at the cost of another).

{\bf Separation of Generalization and Optimization.}
Optimization and generalization are difficult to disentangle in Deep Learning, but our
observations point to a possible separation of concerns.

CG provides a uniform {\em first-order} explanation of memorization and generalization based on the simple observation that each step of SGD preferentially reduces the loss on multiple examples if such directions exist, i.e., coherence (locally at a step in training) leads to generalization (locally at that step). But CG does not explain where the coherence comes from, other than to say it depends on the dataset and the model.

Our experiments show that SGD on neural networks, not just exploits coherence, but creates it. Since this creation happens even with random labels where there is nothing to learn (i.e., no generalization), there is reason to believe that this creation is purely an optimization phenomenon.
Going back to the analogy with random forests, the creation of coherence is similar to the finding of commonality (possibly spurious) between examples during decision tree construction.

Therefore, we believe, what is required is to augment the first-order understanding of generalization provided by CG with a {\em second-order} theory of optimization that explains how coherence or gradient alignment is created.
To that end, the dramatic difference in growth rate of coherence between real and random labels suggests a compounding effect that amplifies existing coherence (perhaps similar in spirit to Lemma~\ref{lem:scale} and mini-batch amplification).
Understanding this process is an important area of future work for us.

\subsubsection*{Acknowledgments}

We thank Sergey Ioffe, Firdaus Janoos, and Alan Mishchenko for many interesting and stimulating discussions on this topic. We thank Michele Covelle, Shankar Krishnan, and Rahul Sukthankar for reviewing early drafts of this paper.


\section{Introduction}

To construct mathematical models or theories corresponding to elements of the physical world is one of the main goals of scientists in general and the physicist in particular. In this process, a theoretical construction is considered more fundamental the greater the class of phenomena it is able to describe. Of all models used to describe the physical world, those based on the space-time concept are among the most successful ones, even though some important questions are still open. One of these is the adequate definition of the quantum relativistic spin operator \cite{Frenkel1926,Thomas1927,Bargmann1935,Lubanski1942,Pryce1948, Fleming1965(1), Wouthuysen1950, Czachor1997, Peres2002, Terno2003, Peres2004, Czachor2003, Polyzou2012, Saldanha2012(1), Caban2013, Taillebois2013, Bauke2014,Giacomini2019}.

The spin concept having been discussed for the first time almost a century ago \cite{Frenkel1926,Thomas1927,Bargmann1935}, one might think that everything important concerning this subject has already been said. However, the description of this concept in a fully relativistic scenario is still an open issue with active discussions, several proposals concerning the definition of a relativistic spin operator and its consequences for setups involving the relativistic measurement of spin having been presented over time. In this sense, we highlight, without the pretension of exhausting the matter, the contributions made by Lubański \cite{Lubanski1942} (with its 4-vector description which embodies the idea of internal angular momentum, in spite of difficulties concerning the connection to real measurement setups), Pryce and Fleming \cite{Pryce1948,Fleming1965(1)} (which made explicit the ambiguities that arise when a relativistic position approach is adopted to describe the system’s spin), Foldy and Wouthuysen \cite{Wouthuysen1950} (with their single-particle approach starting from Dirac’s equation), Czachor \cite{Czachor1997} (who presented the first results concerning relativistic quantum information theory), Peres and Terno \cite{Peres2004} (which demonstrated the apparent non-covariance of spin reduced density matrices), Vedral and Saldanha \cite{Saldanha2012(1)} together with Palmer et al. \cite{Palmer2012} (that approached the problem through a relativistic description of a Stern-Gerlach measurement) and ourselves \cite{Taillebois2013} (with an effective approach to spin based on the fiber bundle character of the relativistic momentum-spin Hilbert space). More recently, we emphasize the contribution from Giacomini et al. in \cite{Giacomini2019}, where the recent formalism of relativistic quantum references frames \cite{Giacomini2019n} has been used to address the problem. On the one hand, this problem is relevant to the foundations of physics, since it is directly connected to the problem of relativistic particle localization \cite{Hegerfeldt1974,Zuben2000,Caban2014,Celeri2016}. On the other hand, solving it may be useful to a wide range of research fields, such as relativistic quantum information theory \cite{Gingrich2002,Caban2005,Caban2006,Landulfo2009,Debarba2012,Saldanha2012(2)}, quantum spintronics \cite{Awschalom2013}, and light-matter interaction at relativistic intensities \cite{Piazza2012, Ahrens2012}.

Here it is argued that the origin of the relativistic spin operator controversy is the lack of a satisfactory definition of what is meant by intrinsicality. Thus, despite the question we have to eventually answer be ``Which spin operator is actually connected to a realistic relativistic spin measurement?", a theoretical approach of what is an intrinsic property in special relativity is exposed first. Once this is done, a unique intrinsic relativistic spin operator and its connection to a realistic experimental setup are presented. This approach allows to shed light over the properties that a spin observable has to satisfy and some misconceptions concerning intrinsicality.

The article is organized as follows.  In Section \ref{sec:Int} we introduce the concept of intrinsicality, its mathematical implementation being formalized for the particular case of quantities defined in the context of special relativity. Assuming the intrinsic character as fundamental for an adequate definition of the relativistic spin concept, in Section \ref{sec:Spin} it is shown that three-vector descriptions of such a quantity must be ruled out. Besides that, in this same section, intrinsicality is also used to define a unique satisfactory intrinsic relativistic spin observable, a consistent observer-independent model for the electromagnetic-spin interaction being presented in Section \ref{sec:Elec}. Finally, final remarks and conclusions are presented in Section \ref{sec:Conc}.

\section{\label{sec:Int}Intrinsicality}

In general the definition of a physical property, such as position or momentum, may be split into three stages. First, the concept of interest $\mathcal{A}$ must be defined, introducing what the property should represent. After this, it is necessary to associate an appropriate mathematical structure $M(\mathcal{A})$ to the concept. The last step is the description: it is necessary to quantify the mathematical structure. This is usually done by introducing a basis $\mathcal{O}$ associated to an observer that measures the property, the description being denoted here by $D(M(\mathcal{A}), \mathcal{O})$. Thus, a specific concept $\mathcal{A}$ leads to a mathematical structure $M(\mathcal{A})$, that is quantified in a basis $\mathcal{O}$ by a description $D(M(\mathcal{A}), \mathcal{O})$.

It is worth nothing that, although the description in general depends on the observer, the mathematical structure may also be observer dependent, as pointed by Fleming \cite{Fleming1965(1)} in the context of quantum relativistic position operators. A simple example of a property with such a behavior is the orbital angular momentum of a set of particles in classical mechanics: if the origin of the system of coordinates is changed, the mathematical structure that describes this property also changes, implying that this property is both system and observer dependent. The independence of the mathematical structure relative to the observer is at the heart of the intrinsic character of a property.

The mathematical definition of a physical property in Minkowski space-time is presented in what follows, Penrose's abstract index tensor notation \cite{Malcolm} being used whenever the introduction of a basis is unnecessary.

\subsection{Physical property in Minkowski space-time}

Suppose that an observer of 4-velocity $v^{a}(t)$, where $t$ is his proper time, wants to analyze some property $\mathcal{A}$ of a classical system at a given instant $t_{0}$. A non-trivial lecture of this statement is that the measurement is being done relative to a space-like hyperplane $v^{\perp}$ that intercepts the observer world-line at a point $P$ where his 4-velocity is $v^{a}(t_{0}) \equiv v^{a}$. If an arbitrary space-time point $O$ is chosen as reference, the later statement implies that the property is being defined relative to an hyperplane given by
\begin{equation}
x^{a}v_{a} = -c^{2}\tau,
\label{hiper}
\end{equation}
$\tau$ being an invariant parameter that defines the position of $v^{\perp}$ relative to $O$ (FIG. \ref{Fig1}). Since the mathematical structure $M(\mathcal{A})$ associated by the observer to the property $\mathcal{A}$ may depend on the space-like hyperplane of measurement and the location of $P$ in that hyperplane, from now on it will be denoted by $M(\mathcal{A}) \equiv A(v^{a},\tau,y^{b})$, $y^{b}$ being the position four-vector of $P$ relative to $O$.

If a second observer, whose world-line also passes by $P$, decides to measure this same property, the mathematical structure that he will associate to the property is going to be related, in general, to an hyperplane that depends on his 4-velocity $v^{\prime a}$ at $P$ (FIG. \ref{Fig2}). Thus, the property as seen by the second observer is related to the hyperplane $v^{\prime \perp}$ given by
$x^{a}v^{\prime}_{a} = -c^{2}\tau^{\prime},$
and the associated mathematical structure is of the form $A(v^{\prime a}, \tau^{\prime}, y^{b})$.
\begin{figure}[ht]
\centering
\includegraphics[scale=0.8]{hyperplane.pdf}
\caption{\footnotesize{Measurement hyperplane $v^{\perp}$ related to an observer of instantaneous 4-velocity $v^{a}(t_{0}) = v^{a}$. The hyperplane is composed by all events that satisfies the relation $x^{a}v_{a} = -c^{2}\tau$, $x^{a}$ describing their position 4-vectors and $\tau$ an invariant parameter that gives the position of the hyperplane relative to the origin $O$. The point $P$ of intersection between the observer world line and the measurement hyperplane is described by the position 4-vector $y^{a}$.}}
\label{Fig1}
\end{figure}
\begin{figure}[ht]
\centering
\includegraphics[scale=0.85]{homogeneous1.pdf}
\caption{\footnotesize{Measurement hyperplanes associated to different observers that pass by the event $P$ described by the position 4-vector $y^{a}$ relative to the event of reference $O$. The first (second) observer of instantaneous 4-velocity $v^{a}$ ($v^{\prime a}$) defines a property relative to the hyperplane $v^{\perp}$ ($v^{\prime \perp}$) that is fixed by the observer 4-velocity and the invariant parameter $\tau$ ($\tau^{\prime}$).}}
\label{Fig2}
\end{figure}

Once established this behavior for a general mathematical structure associated to an arbitrary property, the concept of intrinsicality can be introduced. A property of a system is intrinsic if the related mathematical structure does not depend on the observer that is measuring it (although the description of the property may depend on the choice of a specific observer dependent basis). Thus, the definition of an intrinsic property in special relativity is associated to the dependence of the mathematical structure on the hyperplane related to which it is defined: the mathematical structure of an intrinsic property cannot depend on the orientation of the hyperplane on which it is measured (punctual Lorentz invariance) or on the point of intersection $P$ of the observer world-line with the measurement hyperplane (space-like translational invariance).

\subsection{Space-like translational invariance}

An arbitrary infinitesimal variation of the observer's world line produces a change in $A(v^{a},\tau,y^{b})$ given by
\begin{equation*}
\delta A(v^{a},\tau,y^{b}) = \left(\delta v^{c}\frac{\partial}{\partial v^{c}} + \delta \tau\frac{\partial }{\partial \tau} + \delta y^{c}\frac{\partial}{\partial y^{c}}\right)A(v^{a},\tau,y^{b}).
\end{equation*}
To establish the consequences of space-like translational invariance, an infinitesimal translation of $P$ inside $v^{\perp}$ is considered. This corresponds to setting $\delta v^{c} = 0$ and $\delta \tau = 0$, which leads the constraint \eqref{hiper} to $v_{c}\delta y^{c} = 0$, resulting in
\begin{equation}
\delta A(v^{a},\tau,y^{b}) = \left(\frac{\partial A(v^{a},\tau,y^{b})}{\partial y^{c}} + \lambda v_{c}\right)\delta y^{c} = 0,
\label{link1}
\end{equation}
where $\lambda$ is a Lagrange multiplier introduced to take into account the constraint. Solving \eqref{link1} for $\lambda$ implies that
\begin{equation}
\left(\frac{\partial}{\partial y^{c}} + v_{c}\frac{v^{d}}{c^{2}}\frac{\partial}{\partial y^{d}}\right) A(v^{a},\tau,y^{b}) = 0
\label{vinc1}
\end{equation}
for a property to be invariant under observer translations inside $v^{\perp}$.

\subsection{Punctual Lorentz invariance}

For the case of punctual Lorentz invariance relative to $P$, $\delta y^{c} = 0$ and the hyperplane constraint \eqref{hiper} implies that $y_{c}\delta v^{c} = -c^{2} \delta \tau$. In addition, this kind of transformation does not change $v^{c}$ in an arbitrary way, since the relation $v_{c}v^{c} = -c^{2}$ cannot be violated. Then, the invariance condition must be written as
\begin{eqnarray}
\delta A(v^{a},\tau,y^{b}) & = & \left(\frac{\partial A(v^{a},\tau,y^{b})}{\partial v^{c}} + \lambda_{1}y_{c} + \lambda_{2}v_{c}\right)\delta v^{c} \nonumber \\
& + & \left(\frac{\partial A(v^{a},\tau,y^{b})}{\partial \tau} + \lambda_{1}c^{2}\right)\delta \tau = 0,
\label{link2}
\end{eqnarray}
where $\lambda_{1}$ and $\lambda_{2}$ are Lagrange multipliers introduced to take into account the hyperplane and 4-velocity norm invariance. Solving \eqref{link2} for $\lambda_{1}$ and $\lambda_{2}$ leads to
\begin{equation}
\left(\frac{\partial}{\partial v^{c}} + \frac{v_{c}v^{d}}{c^{2}}\frac{\partial}{\partial v^{d}} - \frac{(y_{c} - \tau v_{c})}{c^{2}}\frac{\partial}{\partial \tau}\right) A(v^{a},\tau,y^{b}) = 0
\label{vinc2}
\end{equation}
for a property to be independent of the measurement hyperplane orientation.

\subsection{Intrinsic physical property in Minkowski space-time}

Relations \eqref{vinc1} and \eqref{vinc2} describe, respectively, the space-like translational invariance and the punctual Lorentz invariance. However their mutual implications also must be taken into account. Performing a punctual Lorentz variation in \eqref{vinc1} and taking \eqref{vinc2} into account results in $A(v^{a},\tau, y^{b}) \equiv A(v^{a}, y^{b})$, which allows to rewrite \eqref{vinc2} as
\begin{equation}
A(v^{a} + \delta v^{a},y^{b})\Big|_{v_{a}\delta v^{a} = 0} = A(v^{a},y^{b})
\label{res1}
\end{equation}
On the other hand, performing a space-like translation variation in \eqref{res1} and taking \eqref{vinc1} into account, it follows that
\begin{equation}
A(v^{a},y^{b}) \equiv A(v^{a})
\label{trans}
\end{equation}
and Equation \eqref{res1} may be rewritten as
\begin{equation}
A(v^{a} + \delta v^{a})\Big|_{v_{a}\delta v^{a} = 0} = A(v^{a}).
\label{res4}
\end{equation}
Equation \eqref{trans} states that to be intrinsic the property must be invariant under any arbitrary translation of the point of observation, while equation \eqref{res4} states the invariance under arbitrary Lorentz transformations related to an arbitrary point of reference. Thus, a relativistic intrinsic property must be associated to the same mathematical structure independent of the observer that is performing the measurement.

\section{\label{sec:Spin} Intrinsicality applied the problem of a relativistic spin operator}

Starting from the framework of intrinsicality, the problem of finding a satisfactory intrinsic spin operator can now be addressed. However, it is instructive to analyze first if there are proposals of relativistic spin operators present in literature that can be ruled out by assuming that the intrinsic character must be fundamental for an observable to be called spin.

\subsection{Non-intrinsic relativistic spin operator}

It is usual to require that to be a spin observable an operator must be a 3-vector, and some of the most important relativistic spin proposals present in literature satisfy this condition \cite{Terno2003,Peres2002,Peres2004,Czachor1997,Giacomini2019}. However, in special relativity a 3-vector can be defined only if a particular space-like hyperplane is specified. Thus, if all observers that pass through a space-time point $P$ define a property as being a 3-vector, the mathematical structure will be observer dependent, since each observer may be associated to a different space-like measurement hyperplane (FIG. \ref{Fig3}). This indicates that a property that is always represented as a 3-vector quantity by any observer cannot be an intrinsic property. A formal deduction of this statement is presented in what follows.

\begin{figure}[ht]
\centering
\includegraphics[scale=0.9]{three_vectors.pdf}
\caption{\footnotesize{Dependence of a 3-vector property on the measurement hyperplane. If $\mathcal{A}$ is a property that is described as a 3-vector by any observer, then the associated mathematical structure will be observer dependent. For an observer of 4-velocity $v^{b}$ it will be given by $A^{a}(v^{b}) \in v^{\perp}$ while for an observer of 4-velocity $v^{\prime b}$ it will be given by $A^{a}(v^{\prime b}) \in v^{\prime \perp}$.}}
\label{Fig3}
\end{figure}

A 3-vector property defined by an observer of 4-velocity $v^{a}$ can be represented by a 4-vector structure $A^{a}(v^{b})$ that satisfies
\begin{equation}
A^{a}(v^{b})v_{a} = 0.
\label{oi1}
\end{equation}
Since it is assumed that any observer passing by the measurement point associates a 3-vector to the property, relation \eqref{oi1} must hold for any four-velocity, i.e.
\begin{equation}
A^{a}(v^{b} + \delta v^{b})(v_{a} + \delta v_{a})\Big|_{v^{b}\delta v_{b} = 0} = 0.
\label{oi2}
\end{equation}
For such a property to be intrinsic it must satisfy \eqref{res4} and, using this together with \eqref{oi1} and \eqref{oi2}, lead to
\begin{equation}
A^{a}(v^{b})\delta v_{a}\Big|_{v^{b}\delta v_{b} = 0} = 0.
\label{oi3}
\end{equation}
Hence, equations \eqref{oi1} and \eqref{oi3} impose that $A^{a}(v^{b}) = 0$, implying that only the null 3-vector can be associate to an intrinsic relativistic property. This result has strong implications, since it invalidates the use of three-vector operators as spin observables if this property is expected to be intrinsic. Besides that, it makes explicit that the problem of non covariance of spin observables present in the relativistic quantum information theory literature \cite{Czachor1997,Peres2002,Terno2003,Peres2004,Landulfo2009} is in fact a consequence of the  non intrinsicality of the observables considered there.

\subsection{Intrinsic relativistic spin operator}

Since three-vectors cannot define an intrinsic property in relativity theory, an important question is left open: is there in fact an intrinsic relativistic spin operator? To answer it, the framework of irreducible unitary representations of the Poincar\'{e} group \cite{Wigner} will be adopted. The advantage of using this formalism is that the results are not restricted only to spin-half particles, being valid for any spin value compatible with the Poincar\'{e} symmetry. A thorough explanation of how this formalism is connected to Dirac spinor formalism for spin-half particles can be found in \cite{Caban2005}.

Quantum mechanically the mathematical structure associated to a property is an observable operator and, on the basis of the correspondence principle \cite{Fleming1965(1)}, the classical results present in \eqref{trans} and \eqref{res4} can be mapped to quantum mechanics. As a result, a property that is intrinsic in relativistic quantum mechanics must be associated to a mathematical structure given by an operator $A(v^ {a})$ that satisfies
\begin{equation}
\mathrm{Tr}\left(\rho A(v^{a})\right) = \mathrm{Tr}\left(\rho^{\prime} A(v^{a} + \delta v^{a})\Big|_{v_{a}\delta v^{a} = 0}\right),
\label{intrinsic1}
\end{equation}
$\rho$ and $\rho^{\prime}$ being the density matrices of the system as described by the two different observers. In the same way, the result concerning the impossibility of using 3-vectors to describe relativistic intrinsic properties also applies to observables in relativistic quantum mechanics.

To determine if there is an intrinsic spin observable, it is necessary to define primarily what is the concept that this property must describe. This is done by imposing a set of three physical reasonable assumptions, the first of them being intrinsicality. Since the spin concept is widely understood as the intrinsic angular momentum of the system, it is natural to define this quantity as being the intrinsic part of the second rank antisymmetric tensor $J_{ab}$ that describes the relativistic angular momentum concept. This construction allows to split the total angular momentum $J_{ab}$ into an orbital $L_{ab}$ and a spin $S_{ab}$ part, both being second rank antisymmetric objects: $J_{ab} = L_{ab} + S_{ab}$. Since $S_{ab}$ must be intrinsic, Equation \eqref{intrinsic1} leads to
\begin{equation}
\begin{split}
\left[ J_{\alpha\beta}, S_{\mu\nu} \right] & = 4i\eta_{[\alpha[\nu}S_{\mu]\beta]}, \\
\left[ P_{\alpha}, S_{\mu\nu} \right] & = 0.
\end{split} \label{eq:dfg}
\end{equation}
These results mean that intrinsicality together with the second rank character imply that $S_{ab}$ must transform as a second rank tensor under arbitrary Poincar\'{e} transformations.

Secondly, the 1-form operator defined by
\begin{equation*}
\tensor[^*]{S}{_{ab}}P^{b} = -\frac{1}{2}\varepsilon_{abcd}S^{cd}P^{b}
\end{equation*}
must coincide with $W_{a} = \tensor[^{*}]{J}{_{ab}}P^{b}$, the so-called Pauli-Lubanski operator. To understand that assumption lets analyze the angular momentum operator $J_{ab}(p)$ for a particle of definite momentum $p^{a}$. For an inertial observer $\mathcal{O}_{v}$ of 4-velocity $v^{a}$, the information contained in $J_{ab}(p)$ can also be retrieved using the pair of 1-forms
\begin{subequations}
\begin{eqnarray}
J_{a}(p;v) & = & \tensor[^*]{J}{_{ab}}\frac{v^{b}}{c} \label{eq2} \\
K_{a}(p;v) & = & J_{ab}\frac{v^{b}}{c}, \label{eq1}
\end{eqnarray}
\end{subequations}
their connection with $J_{ab}(p)$ being given by
\begin{equation}
J_{ab}(p) = \frac{2}{c}v_{[a}K_{b]}(p;v) + \frac{1}{c}\varepsilon_{abcd}v^{c}J^{d}(p;v).
\label{oi10}
\end{equation}
The operators $J_{a}(p;v)$ and $K_{a}(p;v)/c$ are interpreted, respectively, as the total angular and mass momentum observables for a particle of 4-momentum $p^{a}$ as seen by the observer $\mathcal{O}_{v}$, and
\begin{equation}
K_{a}(p;v)v^{a} = J_{a}(p;v)v^{a} = 0. \label{eq10}
\end{equation}
Assuming now that the inertial observer has the same 4-velocity of the particle, i.e. $v^{a} = p^{a}/m$, where $m$ is the particle's mass, equations \eqref{eq2} and \eqref{eq1} can be written as
\begin{subequations}
\begin{eqnarray}
J_{a}(p;p/m) & \equiv & J_{a}(p) = \tensor[^*]{J}{_{ab}}\frac{p^{b}}{mc} \label{eq4} \\
K_{a}(p;p/m) & \equiv & K_{a}(p) = J_{ab}\frac{p^{b}}{mc}. \label{eq3}
\end{eqnarray}
\end{subequations}
Comparing \eqref{eq4} with the Pauli-Lubanski operator and using the interpretation presented before, $J_{a}(p) = \frac{W_{a}(p)}{mc}$ is the total angular momentum of the particle as seen by a co-moving observer. From non-relativistic quantum mechanics it is known that the property called \textit{spin} must be related to the total angular momentum of the particle in its rest frame, thus, assuming that $S_{ab}(p)$ is the extension of this concept, it is expected that the angular part of $S_{ab}(p)$ taken in the rest frame of the particle possesses this interpretation, such that $W_{a}(p) = \tensor[^{*}]{S}{_{ab}}(p)p^{b}$. Extending this to an arbitrary momentum state leads to the second assumption that
\begin{equation}
W_{a} = \tensor[^{*}]{S}{_{ab}}P^{b}.
\label{cond10}
\end{equation}

However, since $W_{a}/mc$ does not contain the full information presented in $S_{ab}$, it is necessary to define the extra component $S_{ab}P^{b}/mc$ to fully determine $S_{ab}$. To do so, a last assumption is made: the intrinsic spin observable must be consistent in the non-relativistic limit, implying that it must lead to a single 3-vector angular momentum quantity in that limit. This means that in the rest frame of the particle it is expected that only the angular momentum contribution of $S_{ab}$ must persist. This, together with \eqref{eq10}, imposes that
\begin{equation}
S_{ab}P^{b}/mc = 0.
\label{cond2}
\end{equation}
Due to the tensor character of $S_{ab}$, which derived from the imposition of intrinsicality, the same construction presented in \eqref{oi10} can be used for $S_{ab}$ and, using \eqref{cond10} and \eqref{cond2}, leads to
\begin{equation}
S_{ab} = \frac{i}{P^{2}}[W_{a},W_{b}].
\label{intspin}
\end{equation}
Equation \eqref{intspin} corresponds to the unique observable that is intrinsic and satisfies the set of minimal conditions presented to define what it is expected from a spin observable. Besides that, $S_{ab}S^{ab}/2 = s(s+1)$, where $s$ is the spin of the particle, i.e. the usual spin norm is frame invariant.

To fully understand the implications of assuming that \eqref{intspin} is the only satisfactory description of an intrinsic spin observable, lets decompose this operator in the 1-form observer dependent description. For an inertial observer of 4-velocity $v^{a}$, the information contained in \eqref{intspin} can be described by the couple of observables $\Sigma_{a}(v) = \tensor[^*]{S}{_{ab}}v^{b}/c$ and $M_{a}(v) = S_{ab}v^{b}/c$. The observables $\Sigma_{a}(v)$ and $M_{a}(v)$ correspond, respectively, to the angular and mass momentum part of $S_{ab}$ as seen by an observer of 4-velocity $v^{a}$. It is important to stress that separately these quantities are not intrinsic properties, since the information that each one carries is observer dependent.

Using a basis ($e^{a}_{(\alpha)}$) with $e^{a}_{(0)} = v^{a}/c$, observable $\Sigma_{a}(v)$ can be written as a 3-vector quantity $\mathbf{\Sigma}(v)$ given by
\begin{equation*}
\mathbf{\Sigma}(v) = \frac{1}{mc}\left(P^{0}\mathbf{S} - \frac{(\mathbf{S}\cdot \mathbf{P})\mathbf{P}}{mc + P^{0}}\right)
\end{equation*}
where $\mathbf{S}$ is the usual Wigner spin operator \cite{Peres2002,Peres2004}. Although the observable $\mathbf{\Sigma}(v)$ transforms as a 3-vector under the action of rotations inside $v^{\perp}$, it does not satisfy the spin algebra unless $v^{a}$ is a co-moving observer. This fact could be used to argument that $S_{ab}$ is not an adequate spin observable, however this argument is misleading and can be circumvented. The requirement of the spin algebra comes from the fact that it is expected that the spin observable be a generator of the rotation group, thus satisfying the $\mathfrak{su}(2)$ algebra. However, if spin is defined as fundamentally being the intrinsic part of $J_{ab}$, the angular momentum character being imposed only in the rest frame of the particle, there is no reason to state that the $\mathfrak{su}(2)$ algebra must be satisfied in an arbitrary relativistic frame of reference other than the particle rest frame. In fact this behavior is even expected since, except if a co-moving observer is being considered, there is not a passive symmetry transformation that rotates only the particle's spin relative to the observer. This schism between the generator and observable character is not new in physics, being present for example in the splitting of electromagnetic angular momentum operator into spin and orbital parts \cite{Enk1994}.

\section{\label{sec:Elec} Electromagnetic coupling with intrinsic spin}

As pointed at the beginning, a theoretical definition of a relativistic spin observable is incomplete if it is not connected to a realistic relativistic spin measurement. Thus, to complete the theoretical argument given above, a simple model of coupling between $S_{ab}$ and the electromagnetic field tensor $F^{ab}$ is presented. The simplest description for this coupling is given by the interaction Hamiltonian
\begin{equation}
\mathcal{H}_{I} = -\alpha \frac{F^{ab}S_{ab}}{2},
\label{hi}
\end{equation}
where $F^{ab}$ is the classical electromagnetic tensor and $\alpha$ is the gyromagnetic ratio. This is in agreement with what was proposed in \cite{Saldanha2012(1),Saldanha2012(2)}, since \eqref{hi} can be rewritten as $\mathcal{H}_{I} = -\alpha B^{a}(P)W_{a}/mc$, where $B_{a}(P) = \tensor[^*]{F}{_{ab}}P^{b}/mc$ is the operator that gives the magnetic field in the rest-frame of the particle, i.e. the interaction is given by the usual spin-magnetic coupling in the instantaneous rest frame of the particle. Equation \eqref{hi} can also be written in observer's frame as
\begin{equation}
\mathcal{H}_{I} = -\alpha \left[\frac{(\mathbf{E}\times\mathbf{P})\cdot\mathbf{S}}{mc^{2}} + \left(\frac{P^{0}}{mc}\mathbf{S}_{\perp} + \mathbf{S}_{\parallel}\right)\cdot\mathbf{B} \right],
\label{spinfield}
\end{equation}
$\mathbf{E}$ and $\mathbf{B}$ being the electric and magnetic fields in this frame, while $\mathbf{S}_{\parallel} = \mathbf{S}\cdot\mathbf{P}/\|\mathbf{P}\|$ and $\mathbf{S}_{\perp} = \mathbf{S} - \mathbf{S}_{\parallel}$ are the parallel and perpendicular components of $\mathbf{S}$ relative to $\mathbf{P}$. Considering a particle of definite 4-momentum $p^{a}$ in non-relativistic limit, equation \eqref{spinfield} can be rewritten as
\begin{equation}
\mathcal{H}_{I} = -\alpha\left[\frac{(\mathbf{E}\times\mathbf{p})\cdot\mathbf{S}}{mc^{2}} + \mathbf{S}\cdot\mathbf{B}\right].  \label{eq:gui}
\end{equation}
The second term in \eqref{eq:gui} is just the usual magnetic-spin interaction, while the first term is just twice the usual spin interaction of the moving electron with the electric field. The fact that \eqref{hi} has led to twice the correct electric interaction term is expected since the Thomas correction \cite{Thomas1926}, that gives a careful treatment for the contribution of electron's acceleration to the electric interaction, was not taken into account. If such contribution is introduced in the interaction Hamiltonian, the above prescription gives exactly the expected electromagnetic spin interaction in the non-relativistic limit, supporting the use of \eqref{intspin} as the adequate intrinsic spin operator.

\section{\label{sec:Conc} Conclusions}

To summarize, it was shown that the use of three-vectors for the description of relativistic spin is not consistent whit the usual conception that this quantity must be associated to the intrinsic angular momentum of the system, the problem of the so-called non covariance of spin observables \cite{Czachor1997,Peres2002,Terno2003,Peres2004,Landulfo2009} being in fact a consequence of the  non intrinsicality of three-vector observables. Then, starting from the intrinsicality condition, it was shown that there is a unique relativistic spin operator that satisfies the properties expected from the concept of an intrinsic angular momentum quantity.

As a final result, it should be noted that the spin observable \eqref{intspin} obtained by imposing intrinsicality as a fundamental property leads to the point-like center-of-inertia position operator \cite{Pryce1948,Fleming1965(1)}. Indeed, this is remarkable since the center-of-inertia of a single particle is associated to its mass, a property that also satisfies the constraints imposed by intrinsicality. This raise an interesting question concerning the definition of intrinsic properties and the localization problem: are the intrinsic properties of a system always related to the same position operator? The answer to this question is left open, a thorough analysis of it being postponed to a future work.

The authors are thankful for the support provided by the Brazilian agencies CAPES (PROCAD2013), FAPEG (PRONEX \#201710267000503, PRONEM \#2017102670-00540), CNPq (\#459339/2014-1, \#312723/2018-0) and the Instituto Nacional de Ciência e Tecnologia - Informação Quântica (INCT-IQ). We also thank Flaminia Giacomini for bringing references \cite{Giacomini2019,Giacomini2019n} to our attention.