\section{Introduction}
When acquiring labeled training data for a budget-constrained image classification application, there is a trade-off between quantity and quality of the data.
Combined annotations from multiple domain experts can be considered correct, but are very expensive.
Data from automatic web crawling or volunteers is more cost-effective but noisy.

Learning from noisy training data has thus been an important research topic.
However, most work approaches label noise from only one angle.
Labels are either correct or not, in which case they are confused with another label. The additional component of label \emph{precision} is usually not taken into account.
For example, a person might annotate an image of a North American barn swallow as simply a perching bird. This label is not inaccurate since it is correct, but it is imprecise.
Learning from such an imprecise label is a case of weakly supervised learning, which generalizes the trade-off between annotation quality and quantity.
However, unlike training labels, all predictions are expected to be precise.
In real-world applications, an output of \enquote{bird} is not helpful to the end user.
We visualize this scenario in \cref{fig:teasertop}.

\begin{figure}
	\centering
	\includegraphics{figures/teaser.pdf}
	\caption{Learning from imprecise data. Our method CHILLAX is capable of fully utilizing training data where labels are imprecise, \emph{i.e.,}\xspace not labeled as leaf nodes of a given class hierarchy. Predictions are always precise.}
	\label{fig:teasertop}
\end{figure}

In this work, we investigate the implications of imprecisely labeled data.
Because standard softmax classifiers are incompatible with imprecise labels, we propose our own method CHILLAX (Class Hierarchies for Imprecise Label Learning and Annotation eXtrapolation).
It is based on hierarchical classification.
This choice has two benefits:
\begin{itemize}
	\item The classifier can utilize all of the available data.
	      Conventional softmax classifiers can only learn from examples labeled at leaf nodes of the class hierarchy, because the rest violates their assumption of mutually exclusive classes. Such violations occur in large web crawled datasets like ImageNet~\cite{deng_imagenet:_2009}. It contains the concepts \enquote{electronic device} and \enquote{video display} which each consist of some unique images but are not mutually exclusive.
	\item We integrate domain knowledge in the form of a class hierarchy directly into the classification process.
	      This additional information can improve accuracy and learning speed \cite{brust_integrating_2019}.
\end{itemize}

To evaluate the performance of our method, we build upon on the well-known benchmark datasets NABirds \cite{van_horn_building_2015} and ILSVRC2012 \cite{russakovsky_imagenet_2015}.
We reduce the label precision of this data synthetically to simulate imprecisely labeled data.
The reduction is performed using statistical models for label precision depending on the specific data source.
One example is web crawling for training data, or webly supervised learning, a research area of increasing relevance.
Volunteer annotators such as citizen scientists or hobbyists are considered as well.
We construct the statistical models to reflect the unique characteristics of these data sources.
Furthermore, we validate our noise models on image metadata obtained by web crawling.
This shows that our synthetic data matches the properties of real-world data.

Experiments on this synthetic data compare CHILLAX against strong baselines and the state of the art HEX~\cite{deng_large-scale_2014}.

\section{Related Work}
\label{sec:relatedwork}
While the topic of semantically imprecise labels is not widely considered, there is still relevant literature in closely related fields.
This section gives a brief overview of work concerning models of label noise and hierarchical classification methods.

Label noise is usually described in terms of confused labels. Patrini \emph{et al.}\xspace model label noise as class-conditional \cite{patrini_making_2017}.
Each class has a certain probability of being changed to any other class. These probabilities are defined using a transition matrix.
It could describe confusions caused by similarity, especially in a fine-grained setting.
In \cite{li_learning_2017}, label noise is modeled more generally.
A completely unknown process changes the labels, \emph{i.e.,}\xspace it may not even depend on the ground truth.
In \cite{dehghani2017learning}, label noise is considered in a weakly supervised learning setting.
These works do not consider semantics explicitly.
However, there is evidence that uniformly random confusions are not representative of real-world label noise~\cite{jiang2020noise}, highlighting the importance of our investigation and the relations between labels.
Also, \cite{qin2004tree} offers a semantic perspective on uncertainty as \enquote{fuzzy sets}. Here, the imprecision is not in the annotations, but the class labels themselves.

Semantics are looked at more often for improving learning methods.
For example, \cite{bertinetto_making_2019} discusses the benefits of a hierarchical perspective on classification.
However, most methods that leverage class hierarchies are used for metric learning \cite{hwang_learning_2011,verma_learning_2012,zhang_embedding_2016}, zero-shot learning  \cite{huo_zero-shot_2018,li_zero-shot_2015,rohrbach_transfer_2013,ye_zero-shot_2017}, as a regularizer \cite{fergus_semantic_2010,goo_taxonomy-regularized_2016,srivastava_discriminative_2013}, or to build an embedding space \cite{barz_hierarchy-based_2019,faghri_vse++:_2017,frome_devise:_2013,hwang_unified_2014}.
Deng \emph{et al.}\xspace acknowledge the problem of semantic imprecision as a result of crowdsourced or crawled annotations in \cite{deng_large-scale_2014}.
Yet, their model of label noise is very simplistic as it only looks at the bottom two layers of a class hierarchy consisting of more than ten layers.
It therefore fails to capture the nuances of differently skilled human annotators (see \cref{ssec:dengmodel}).
The notion of label precision is balanced with accuracy on a prediction level in \cite{deng2012hedging}.

We build our method CHILLAX upon the work by Brust \& Denzler \cite{brust_integrating_2019}.
They propose a probabilistic hierarchical classifier to improve accuracy over conventional softmax classifiers.
A probabilistic model encodes common assumptions about class hierarchies.
To apply it to deep classifiers, they derive a label embedding and a task-specific loss function.
In the following, we will show how this method can be adapted to accommodate imprecisely labeled data.

\section{Learning from Semantically Imprecise Data}
\label{sec:sid}
This section details our definition of semantically imprecise data and the construction of our method for fully utilizing it, called CHILLAX.

Let $f: \mathcal{X} \to \mathcal{Y}$ be a classifier with:
\begin{itemize}
	\item $\mathcal{X}$ the input images, \emph{e.g.,}\xspace $\mathbb{R}^{224\times 224}$.
	\item $\mathcal{Y}$ the label set, \emph{e.g.,}\xspace the set of all bird species.
\end{itemize}

We then define the \emph{imprecise} label set $\mathcal{Y}^+$ as the set of all semantic ancestors of $\mathcal{Y}$.
In the example of bird species, the set $\mathcal{Y}^+$ corresponds to regna, phyle, familiae etc.
For better contrast, we call $\mathcal{Y}$ the \emph{precise} label set.

We require the classifier to predict only precise labels from $\mathcal{Y}$.
At the same time, it needs to be able to learn from training data with labels from both $\mathcal{Y}$ and $\mathcal{Y}^+$, which we refer to as semantically imprecise data.
This formulation is a weakly supervised learning scenario.
However, the training set may contain both weak (imprecise) and strong (precise) labels at the same time.

\subsection{Hierarchical Classification}

\begin{figure*}[t]
	\centering
	\subfloat [Original loss mask from \cite{brust_integrating_2019}. This annotation of \enquote{animal} is interpreted as neither \enquote{cat} nor \enquote{dog}. The respective nodes are trained to $p=0$.]{%
		\includegraphics[width=0.40\linewidth]{plots/mdlmaskorig.pdf}
		\label{fig:mdlmaskorig}
	}\hfil
	\subfloat [CHILLAX loss mask. With an imprecise \enquote{animal} annotation, it could be a \enquote{cat} or \enquote{dog}, but it is unknown which. The respective nodes are not trained to reflect this lack of knowledge.]{%
		\includegraphics[width=0.40\linewidth]{plots/mdlmaskmod.pdf}
		\label{fig:mdlmaskmod}
	}
	\caption{Comparison of original loss mask $m$ \cite{brust_integrating_2019} and
		our modification. In this example, the ground truth is \enquote{animal}. Dark nodes are trained and light nodes are not.}
	\label{fig:lossmasks}
\end{figure*}

In this work, we adapt a probabilistic hierarchical classifier proposed by Brust \& Denzler in \cite{brust_integrating_2019}.
It models the semantics associated with \textbf{is-a} relations using the following equation (cf.~\cite[eq.~(5)]{brust_integrating_2019}):
\begin{equation}
	P(Y_s^+|X) = P(Y_s^+|X, Y_{S'}^+)P(Y_{S'}^+|X),
\end{equation}
where $P(Y_s^+|X)$ is the probability of a label $s$ being present given an image $X$.
The event $Y_{S'}^+$ indicates that \emph{any} of the hypernyms of $s$ are present.
If no hypernyms of $s$ are present, then $s$ cannot be present itself,
except for the root.
This equation is recursive, with (cf.~\cite[eq.~(6)]{brust_integrating_2019}):
\begin{equation}
	P(Y_{S'}^+|X) = 1 - \prod_{i=1}^{|S'|}{1-P(Y_{s'_i}^+|X)},
\end{equation}
where $S'$ is the set of hypernyms of $s$.

To apply this model to a neural network, each label (precise and imprecise) is assigned one node in a sigmoid output layer.
A node assigned to $s$ predicts the conditional probability $P(Y_s^+|X, Y_{S'}^+)$.
The targets for each predictor are calculated by a label encoding $e$, which is 1 if a label is present and 0 otherwise.
Binary cross-entropy is used as a loss function.
To train nodes only when the truth value of their respective label is known, a loss mask $m$ is applied~\cite{brust_integrating_2019}.
Its precise definition, and our proposed modification to it, is given in the following.

\subsection{A Loss Mask for Semantically Imprecise Data}
The aforementioned classifier is in principle capable of handling semantically imprecise data.
However, it treats inner nodes as mutually exclusive with their children, assuming that a label is always as precise as possible.
A label \enquote{bird} then means \enquote{a type of bird, but not one present in the hierarchy}.
This is a result of optimizing for a slightly different task than ours.
For CHILLAX, we require \enquote{bird} to only mean \enquote{some type of bird}.

To achieve this, we modify the loss mask $m$ in eq. (12) of \cite{brust_integrating_2019} such that the children of the labeled node are not trained at all.
The original $m: \mathcal{Y}^+ \to \{0,1\}^{|\mathcal{Y}^+|} $ is defined in \cite{brust_integrating_2019} as:
\begin{equation}
	m(y)_s = \begin{cases}
		1 & \text{$y=s$ or}                                                    \\
		  & \exists (s,s') \in h: y = s' \text{ or }(y,s') \in \mathcal{T}(h), \\
		0 & \text{otherwise},
	\end{cases}
\end{equation}
where $s \in \mathcal{Y}^+$ is the component deciding whether the node representing $s$ should be trained, given a label $y$.
$h$ is the hyponymy relation and $\mathcal{T}(h)$ its transitive closure, such that $(s,s')\in h$ means $s$ \textbf{is-a} $s'$ \cite{brust_integrating_2019}.

In other words: given the label $y$, train the $s$ node if:
\begin{enumerate}
	\item $y = s$,
	\item $s$ \textbf{is-a} $y$, or
	\item $s$ \textbf{is-a} any (transitive) hypernym of $y$.
\end{enumerate}

To make the classifier compatible with our interpretation of imprecise labels, we need to remove the second criterion.
This way, the nodes representing the ground truth label's children are not trained, because we do not know their truth values.
We then define our alternative loss mask $m'$ as:
\begin{equation}
	m'(y)_s = \begin{cases}
		1 & \text{$y=s$ or}                                  \\
		  & \exists (s,s') \in h: (y,s') \in \mathcal{T}(h), \\
		0 & \text{otherwise},
	\end{cases}
\end{equation}
removing the second criterion $\exists (s,s') \in h: y = s'$ and only training the nodes where our knowledge is sufficient.
See \cref{fig:lossmasks} for a visualization of the original loss mask and our modification.

Note that changing the encoding $e$ is not necessary. Originally, it encodes children of the ground truth label as negative examples. This assignment does not fit our task, in which we do not know the truth value at all. However, our modified loss mask $m'$ ensures the relevant nodes are not trained either way. Only leaf nodes are evaluated for prediction.

\section{Modeling Semantic Imprecision}
\label{sec:noisemodels}

\begin{figure*}[t]
	\centering
	\subfloat [Web Crawling] {%
		\includegraphics[width=0.23\textwidth]{plots/example-distributions/deng-power-law.pdf}
		\label{sfig:geometric}
	}
	\subfloat[Volunteer] {%
		\includegraphics[width=0.23\textwidth]{plots/example-distributions/poisson.pdf}
		\label{sfig:poisson}
	}
	\subfloat[Relabeling \cite{deng_large-scale_2014}] {%
		\includegraphics[width=0.23\textwidth]{plots/example-distributions/deng-relabeling.pdf}
		\label{sfig:deng}
	}
	\subfloat[Professional / Benchmark] {%
		\includegraphics[width=0.23\textwidth]{plots/example-distributions/benchmark.pdf}
		\label{sfig:bench}
	}
	\caption{The depth of a label in a given class hierarchy represents its precision. The graphs show distributions over label depth as modeled for different annotation sources. Benchmark data does not show any imprecision.}
	\label{fig:example-distributions}
\end{figure*}

We need a more fine-grained approach to label precision in order to study different types of semantically imprecise data properly.
Instead of classifying labels as either precise or imprecise, we define precision as depth in a given class hierarchy.

With this definition, we can model the precision statistics of different data sources.
In the following, we consider data crawled from the web as well as data provided by volunteers.
For reasons explained in the following, both are semantically imprecise.
We contrast these sources with benchmark data that shows no imprecision at all, and related work that models very mild imprecision \cite{deng_large-scale_2014}.

The models are visualized in \cref{fig:example-distributions} and validated on real-world data in \cref{ssec:realworlddata}.

\subsection{Web Crawling}
\label{ssec:webcrawling}
Labels crawled from the web are typically extracted from image captions, hashtags, or accompanying text.
The authors of this information are not labeling purposely, but instead attempting to describe an image for accessibility or encourage sharing and interaction.
Such a process seldom results in a precise label.
In fact, we expect most images to be barely labeled at all:
a caption of \enquote{this beautiful \#bird i saw while jogging} is more likely than \enquote{enjoy this non-breeding bunting!}
There are many more problems with web crawling, such as filtering, or a choice of pre-defined possible labels.
We address some of these challenges in \cref{ssec:realworlddata}, where we validate this model with real-life data.

We model the probability of a label having a certain depth in the hierarchy.
For web-crawled data, we apply a geometric distribution (\cref{sfig:geometric}).
This model is described, but not investigated further in \cite{deng_large-scale_2014}.

Depending on the class hierarchy, it can be useful to offset the distribution by a number of layers.
The offset is such that the first layers have zero probability of occurring, and the distribution is \enquote{shifted to the right}.
This shift models the extreme degree of abstractness in the first layers of hierarchies such as WordNet \cite{fellbaum_wordnet_1998},
which is not useful in practice.
In \cref{fig:realworld}, this shift can be observed.

\subsection{Volunteer Labelers}
\label{ssec:volunteer}
We consider images labeled by volunteers, or for small amounts of money, \emph{e.g.,}\xspace Mechanical Turkers.
Volunteers are aware that they are labeling and annotate to the best of their abilities, but may still not be experts in the problem field.
Only a small fraction can label at the most precise level, but most will be able to provide some information.
Compared to web crawling, it is easier to restrict volunteer annotators to a specific set of labels and images.
This restriction eliminates the problem of mapping annotations to a class hierarchy.
There is also no need to filter the data if images are pre-selected.

How to perform interaction in a manner that results in the highest possible precision is still an open research question.

The combination of a best-effort annotation on the one hand and lack of expertise on the other leads us to assume a Poisson-distributed dataset (\cref{sfig:poisson}).
Shifting the distribution depending on the class hierarchy in question can be useful, as described in \cref{ssec:webcrawling}.

\subsection{Relabeling to Immediate Parents}
\label{ssec:dengmodel}
Deng \emph{et al.}\xspace initially propose this type of noise in \cite{deng_large-scale_2014}.
They use it to evaluate their method HEX (Hierarchy and Exclusion graphs).
Their setup replaces a fixed fraction of precise labels with the direct parent label (cf. \cref{sfig:deng}).
In a deep hierarchy, the precision is only marginally affected.
There is not much information lost by going up only one level,
particularly in a deep hierarchy with low fan-out.
Determining a single parent as replacement is only possible in tree hierarchies, but not all directed acyclic graphs.

\subsection{Professional / Benchmark Datasets}
Professionally created datasets exhibit few or none of the quality problems described in the previous sections.
They are typically acquired by asking paid domain experts for labels, or by running extensive citizen science projects.
Before labeling, the images are carefully selected and controlled for quality.
Multiple annotators label the same image to further improve accuracy by consensus.
This process is the gold standard, where labels are only of the highest precision and accuracy.
However, only a small number of these datasets exist and are available to the general public.
For budget reasons, a high-quality dataset may not be the first choice for many projects.

\Cref{sfig:bench} illustrates this extreme situation.

\section{Experiments and Evaluation}
In this section, we validate our proposed method CHILLAX in three experiments.
We first investigate the effects of imprecision using the models proposed in \cref{sec:noisemodels} in detail.
This controlled experiment is important to isolate the effects.
Then, we add inaccuracy, \emph{i.e.,}\xspace incorrect labels, to better reflect a real-world setting, where both types of label noise will occur.
We also compare CHILLAX with HEX \cite{deng_large-scale_2014} to validate its competitiveness.

\subsection{Setup}
\label{ssec:expsetup}
This section details our experimental setup, including datasets, methods and technical details.
We use the CHIA framework for our experiments. Code
\footnote{\url{https://github.com/cvjena/chia} and \url{https://github.com/cvjena/chillax}}
is publicly available.

\paragraph{Training Data}
There are no publicly available benchmarks specifically for this use case.
Instead, we build synthetic datasets and carefully validate the underlying assumptions.
The North American Birds (NABirds,~\cite{van_horn_building_2015}) and ImageNet Large Scale Visual Recognition Challenge 2012 (ILSVRC2012,~\cite{russakovsky_imagenet_2015}) datasets are used as a starting point.
We degrade the training datasets using the noise models described in \cref{sec:noisemodels}.
The hierarchies that are supplied as part of the datasets are used in all cases to avoid any manual mapping errors.
NABirds offers a tree hierarchy. WordNet, underlying ILSVRC2012, is not a tree as it contains (undirected) cycles.

Our goal is to modify the training datasets such that the distribution over label depths in the hierarchy matches a given noise model.
For each original label, we sample a depth from the model and go up the hierarchy until this depth is reached.
If there are multiple parents to choose from at the targeted depth, one is randomly selected.
There is never a need to go down the hierarchy,
since all datasets are labeled as leaf nodes.
To fit each dataset, all distributions are truncated to not exceed the maximum depth of the given hierarchy.
We apply no shift to the distributions in our synthetic data.

There is a distinct advantage to using synthetic data.
It allows us to control the exact degree of imprecision, highlighting the strengths and weaknesses of each method.

\paragraph{Baselines}
We evaluate our method against strong baselines, using the typical softmax classifier and cross-entropy loss function.
The softmax classifier assumes mutually exclusive classes.
As such, this classifier can only learn from examples labeled as leaf nodes.
However, the degradation process can remove a large fraction of these precise examples.
The resulting imprecise examples cannot be used directly by the softmax classifier.

We propose two baseline strategies for handling the imprecise examples:
\begin{enumerate}
	\item Leaves only: filter out imprecise examples, leaving only a portion of the dataset.
	\item Random leaf: Project imprecise examples to a random descendant leaf node, utilizing the whole dataset, but introducing inaccuracies.
\end{enumerate}

These baseline methods represent the current state of not using imprecise data at all, and the alternative of making an educated guess and turning a weak label intro a strong (precise) one.
Without imprecision, they are identical.
To the best of our knowledge, the only other method capable of performing our task is HEX \cite{deng_large-scale_2014}. However, it cannot be replicated or applied to new datasets with reasonable effort.
We evaluate our method against HEX in \cref{ssec:hex}.
A typical hierarchical classifier would need considerable modifications because of our task's asymmetry of training and prediction label sets.
Since these modifications (specifically the constraint on predicted labels) are not trivial, but have a large influence on the results, we do not use common hierarchical classifiers as baselines.

For better comparison with other works, we report metrics on the respective unmodified validation datasets.

\paragraph{Neural Network Training}
Our deep learning setup matches the ResNet50 described in \cite{brust_integrating_2019}, with a few exceptions.
Our learning rate schedule is SGDR \cite{loshchilov_sgdr_2017}.
There is no data augmentation except for random horizontal flipping.
For ILSVRC2012, our main goal to approximate the performance of \cite{deng_large-scale_2014} when there is no noise,
while our NABirds setup should come closer to the state of the art.
A detailed listing of hyperparameters can be found in \cref{tbl:hyperparams}.

\begin{table}
	\caption{Detailed Hyperparameters for all Experiments.}
	\label{tbl:hyperparams}
	\centering
	\footnotesize{%
		\begin{tabular}{lll}
			\toprule
			Setting            & NABirds                                     & ILSVRC2012         \\
			\midrule
			Max. Learning Rate & 0.003                                       & 0.2                \\
			Min. Learning Rate & $10^{-6}$                                   & $10^{-5}$          \\
			$T_0$              & 80 epochs                                   & 10 epochs          \\
			Warmup             & 1 epoch @ lr=0.01                           & 5 epochs @ lr=0.05 \\
			Training Duration  & 80 epochs                                   & 20 epochs          \\
			Pre-Training       & iNaturalist 2018 \cite{Cui2018iNatTransfer} & ---                \\
			Batch Size         & 32                                          & 128                \\
			Image Size         & $512\times512$                              & $256\times256$     \\
			Random Crop Size   & $448\times448$                              & $224\times224$     \\
			\bottomrule
		\end{tabular}}
\end{table}

\subsection{Imprecision Only}
\label{ssec:expimponly}
We first investigate the effects of imprecise labels in a controlled fashion.
Only the noise model is applied, and there are no other modifications to the training data.
\Cref{tbl:nabirds_accuracy_imprecision_only} shows the results obtained by six runs each of our method CHILLAX and the two baselines.

We observe that CHILLAX's performance relative to the baselines increases with the level of noise applied to the training data.
Without noise, it reaches an accuracy of $81.4\% \pm 0.2$, thus performing slightly worse than the baseline one-hot classifier at $82.8\% \pm 0.2$.

But adding noise, we outperform the baselines in almost all cases, except for the geometric noise model with $q > 0.9$.
We observe substantial improvements in high noise situations such as the Poisson model at $\lambda=1$.

For geometric and Poisson noise, the \enquote{leaves only} baseline shows better results than the \enquote{random leaf} variant.
This is likely because of the strong confusions caused by selecting a random leaf.
The results in the relabeling scenario support this.
Here, selecting a random leaf is beneficial because it is not as likely to be false as in the other scenarios.
From the second to last level in the hierarchy, there are only approximately two child nodes to choose from, instead of the tens or hundreds that are reachable from higher levels, as happens in the geometric and Poisson cases.

The slightly worse accuracy without any noise is possibly due to problematic assumptions about the correlation between visual and semantic similarity \cite{brust_not_2018,brust_integrating_2019}.
This affects CHILLAX negatively, because it is based on hierarchical classification.
But this effect is outweighed by the advantages of learning directly from imprecise data in almost all cases.

\begin{table}
	\centering
	\caption{Accuracy (\%) and standard deviation on the NABirds dataset of our method CHILLAX and the baselines. We evaluate the geometric (web crawling), Poisson (volunteer), and relabeling imprecision noise models. There is no added inaccuracy.}
	\footnotesize{\input{include/table_nabirds_accuracy}}
	\label{tbl:nabirds_accuracy_imprecision_only}
\end{table}

\subsection{Semantic Error Analysis}
\label{ssec:lcaanalysis}
Hierarchical classifiers often exhibit a different quality of errors from regular ones, \emph{i.e.,}\xspace they make \enquote{better} mistakes \cite{bertinetto_making_2019}.
To determine if CHILLAX also benefits from this property, we calculate the depth of the lowest common ancestor (LCA) of each mispredicted label-prediction pair in the validation set.

\Cref{tbl:lcad} shows the LCA depth corresponding to the previously reported results, obtained during the same experiment (see \cref{ssec:expimponly}).
In most cases, the baselines behave as expected.
The LCA depth increases when the labels are more precise, except for the random leaf baseline in the relabeling setting.
There, the LCA depth decreases with more precise labels.
In other words, the baseline makes \enquote{better} mistakes when trained on noisy data.
CHILLAX shows the same unexpected behavior, where the least precise labels always lead to the highest LCA depth.

A possible explanation is that the classifier \enquote{focuses} on the earlier layers of the class hierarchy, simply because the respective labels make up most of the training data in the noisier case.
However, this does not apply to the relabeling case, where label depth is reduced by one layer at most, and we still observe similar results.

Regardless of this anomaly, in most cases, CHILLAX makes the \enquote{best} mistakes from a semantic perspective.

\begin{table}
	\centering
	\footnotesize{%
		\caption{Lowest common ancestor (LCA) depth between ground truth and mispredictions on the NABirds dataset. We evaluate the geometric (web crawling), Poisson (volunteer), and relabeling imprecision noise models. \emph{Higher is better}.}
		\label{tbl:lcad}
		\input{include/table_nabirds_error_depth}}
\end{table}

\subsection{Inaccuracy and Imprecision}
While the previous experiment is important to evaluate our method in a controlled manner, we should also investigate the effects of inaccurate data.
When acquiring labeled data from noisy sources, it is unlikely that it is only imprecise.
Literature predominantly considers inaccuracy when mentioning label noise (see \cref{sec:relatedwork}).
Inaccuracy can result from a lack of knowledge, errors language processing used in web crawling, and can also be intentional as a form of vandalism.

To simulate inaccuracy in this experiment, we replace a fraction of the precise training data with randomly selected labels of the same high precision, \emph{i.e.,}\xspace leaf nodes.
Imprecision is applied afterward, as described in \cref{ssec:expsetup}, thus compounding both types of label noise.
Note that the final fraction of inaccurate annotations can be lower because two confused classes can have overlapping ancestors that are selected by the imprecision process.
Alternatively, one could apply imprecision first. However, applying inaccuracy afterwards is not trivial since there are more options for confusion that can not be treated equally.

\Cref{tbl:nabirds_accuracy_twonoise} shows the effects of randomly confusing $10\%$ of the training data in six runs each.
Without any additional imprecision, the baseline reaches an accuracy of $77.3\% \pm 0.1$, slightly outperforming our method at $75.3\% \pm 0.4$.
Applying the imprecision models, the results are similar to \cref{ssec:expimponly}.
We still observe CHILLAX performing better than the baselines when there is more imprecision, although the amount required for outperformance is slightly elevated.
A $1\%$ confused training data experiment exhibits similar behavior.

CHILLAX is affected disproportionately by inaccuracy label noise.
When the baseline classifier encounters two randomly confused classes, two nodes are trained in the wrong direction.
Applying the same confusion to a hierarchical classifier, two whole paths can be affected.
A wrong decision in an early layer causes mispredictions on a much larger scale.

Despite these fundamental disadvantages of hierarchical classifiers in the face of a high fraction of inaccurate labels, CHILLAX performs very well, and still better than both baselines in most cases.

\begin{table}
	\centering
	\caption{Accuracy (\%) and standard deviation on the NABirds dataset of our method CHILLAX and the baselines. We evaluate the geometric (web crawling), Poisson (volunteer), and relabeling imprecision noise models. $10\%$ inaccuracy is applied.}
	\footnotesize{\input{include/table_nabirds_twonoise}}
	\label{tbl:nabirds_accuracy_twonoise}
\end{table}

\subsection{Comparison to HEX \cite{deng_large-scale_2014}}
\label{ssec:hex}

\begin{figure*}
	\centering
	\includegraphics[width=\textwidth]{plots/per_image_paper.pdf}
	\caption{Imprecision in Flickr: Frequency over depth in the WordNet hierarchy. A most descriptive synset is calculated per image, and separately for the title, description, and tags in the image metadata to highlight the differences.}
	\label{fig:realworld}
\end{figure*}

We do not consider the experimental protocol proposed by Deng \emph{et al.}\xspace in \cite{deng_large-scale_2014} to be representative of our specific task.
Relabeling classes to their immediate parents in a deep hierarchy does not reduce label precision substantially.
This effect can be observed in \cref{ssec:expimponly}, where selecting a random leaf node achieves good performance even when relabeling almost all samples to their parents.
We also visualize the small influence on precision in \cref{fig:example-distributions} (c) and (d).

Still, a comparison with existing methods is an important part of validating a proposed method and the setup described in \cite{deng_large-scale_2014} comes closest to our scenario.
As such, their method HEX can be considered the state of the art.

For a fair comparison, we first tune CHILLAX on ILSVRC2012 to achieve similar performance to HEX when there is no noise.
CHILLAX reaches an accuracy of $62.5\%$ ($83.5\%$ top-5) on the validation set, compared to $62.6\%$ ($84.3\%$ top-5).
This setup means that our method does not have any advantage simply because of a more recent deep neural network architecture, so any differences in performance can be attributed to the different hierarchical classifiers more easily.
We observe the performance of CHILLAX and both baselines under the four noise levels proposed by Deng \emph{et al.}\xspace and compare these results to their HEX.

\begin{table}
	\centering
	\footnotesize{%
		\caption{Top-1 (top-5) accuracy ($\%$) on ILSVRC2012 with relabeling to immediate parent. We evaluate our method CHILLAX against HEX \cite{deng_large-scale_2014} following the same experimental protocol.}
		\input{include/table_deng}}
	\label{tbl:deng}
\end{table}

\Cref{tbl:deng} shows the results of this experiment. At $50\%$ to $90\%$ noise, CHILLAX outperforms HEX in both metrics.
When inspecting top-5 accuracy, it is superior at all noise levels.
However, HEX does achieve a higher top-1 accuracy at $95\%$ and $99\%$.
Both baselines are not competitive in any setting, although selecting random leaves performs surprisingly well at high relabeling fractions.

The comparison between CHILLAX and HEX does not correspond to our earlier findings where our method's performance advantage over the baselines increased with higher noise.
However, HEX is a hierarchical method like ours and very different from the baselines in the previous experiment.

The advantage in top-5 accuracy, even at lower top-1 accuracy, comes from our method of computing per-class probabilities.
The conditional representation means that each predictor can concentrate independently on a straightforward task and does not need to coordinate with other predictors like in a softmax setting.
Sometimes, this missing global view of classification results in a wrong ranking of predictions because predictors are not trained to \enquote{overpower} each other.
This results in a slightly lower top-5 accuracy.
But it increases the probability of predicting less represented classes, leading to higher top-5 accuracy.

Overall, CHILLAX is competitive with HEX, but more generally applicable to data with arbitrary levels of imprecision.

\subsection{Imprecision in Real-World Datasets}
\label{ssec:realworlddata}

To validate the models described in \cref{sec:noisemodels}, we analyze image metadata from Flickr \footnote{\url{https://www.flickr.com/}} as an example.
We obtain approximately $1,500,000$ data points corresponding to images taken from January 1st, 2019 to December 31st, 2019.
Our goal is to map each image to oe synset in WordNet \cite{fellbaum_wordnet_1998} if at all possible.
Note that the first few layers of the WordNet noun hierarchy do not contain concepts that are meaningful descriptions of image content.
Hence, they rarely occur in the image metadata.
Thus, we expect the precision distributions to be slightly \enquote{shifted}.
We also expect a slight bias towards terms used in photography, such as \enquote{aperture}, \enquote{lens} and \enquote{focal length}.
The respective synsets have depths 7 to 10.

The title, description and list of (human-supplied) tags are processed separately for a more detailed analysis.
This information is tokenized and then lemmatized.
The lemmatization tries to find a matching noun because that is the most likely setup for a classification dataset.
For each lemma, there are many possible synsets.
We select the one closest to the root, avoiding over-interpretation.
The result of this process in a set of synsets per image.
To obtain a single label, the synset farthest from the root is selected because we expect it to be most descriptive of the image.
If a synset cannot be determined, we ignore the image altogether.

\Cref{fig:realworld} shows the distribution over depth in WordNet of the most descriptive synset per image.

The distribution obtained from the image title aligns most closely with the volunteer model as described in \cref{ssec:volunteer}.
For the other distributions, the web crawling model (\cref{ssec:webcrawling}) fits well when shifting the distribution to ignore the first three or four (meaningless) layers of the hierarchy.

Title, description and tags exhibit different characteristics because they are used towards contrasting goals.
Titles are a more artistic expression and very descriptive, while tags should be generic to match as many search queries as possible.
The description field rarely contains additional information, if used at all.
It is often a sentence combining the title and tags, or a direct copy of the title.
The distribution (\cref{fig:realworld}) appears to reflect this property.

We consider this real-world data a validation of the models proposed in \cref{sec:noisemodels}.

\section{Conclusion}
In this work, we consider the rarely discussed, but critical task of learning from semantically imprecise data.
Annotation processes such as web crawling or crowd sourcing naturally produce weak labels with imprecision, where we model precision in terms of depth in a given class hierarchy.

Conventional classifiers, \emph{e.g.,}\xspace deep neural networks with softmax outputs learning from one-hot encoded labels, are incompatible with any but the most precise training data.
Because they model classes as mutually exclusive, they will draw wrong conclusions if they encounter imprecise labels.

Our method CHILLAX is capable of weak supervision, \emph{i.e.,}\xspace learning from imprecise and precise labels at the same time.
It always extrapolates to predictions at the highest precision.

To the best of our knowledge, this learning problem is not considered in existing literature.
Only \cite{deng_large-scale_2014} describes a very basic formulation.
Consequently, there are no benchmarks for this problem.
To evaluate our method's performance, we build a synthetic dataset based on a popular benchmark.
We take the North American Birds dataset \cite{van_horn_building_2015} because it ships with a well understood class hierarchy.
Species recognition also represents a typical use of volunteer annotations, where imprecise labels will occur.
The training set is weakened by replacing labels with ancestors of a certain depth, according to our imprecision models (see \cref{sec:noisemodels}).

If synthetic data is to be representative of real-world problems, the underlying assumptions have to be carefully validated.
We do this by analyzing image metadata crawled from Flickr.
The distributions obtained from image titles, descriptions, and tags validate our model assumptions.

Using this real-world validated synthetic data, we show that our method outperforms strong baselines in almost all cases.
On imprecise NABirds modelling a noisy volunteer annotation scenario, CHILLAX achieves an accuracy of $42.9\%$, a substantial improvement to the best baseline at $26.5\%$.
We also compare our method to the state-of-the-art HEX \cite{deng_large-scale_2014} using their ILSVRC2012 experimental protocol and observe superior performance in most settings. At $50\%$ relabeling, CHILLAX reaches an accuracy of $62.1\%$ vs. HEX at $58.2\%$.

\textbf{Future Work} With higher imprecision, a significant amount of images are labeled as the root of a given class hierarchy.
If the fraction of such examples is high, an unsupervised learning method could help increase sample efficiency.

Building a new dataset from real sources is an interesting next step.
Examples of such data sources are discussed in  \cref{sec:noisemodels}.
Several small problems, including the development of an interaction mechanic for imprecise labeling, have to be solved along the way.

\textbf{Acknowledgments}
The computational experiments were performed on resources of Friedrich Schiller University Jena supported in part by DFG grants INST 275/334-1 FUGG and INST 275/363-1 FUGG.

\bibliographystyle{IEEEtran}


\section{Introduction}

A compact K\"ahler manifold admits a holomorphic symplectic form if and only if it
admits a hyperK\"ahler structure \cite{Be}, \cite{Ya}. To explain this, let $X$ be a compact manifold
equipped with almost complex structures $J_1,\, J_2,\, J_3$, and let $g$ be a Riemannian metric on $X$,
such that $(X,\, J_1,\, J_2,\, J_3,\, g)$ is a hyperK\"ahler manifold. Then $g$ defines a $C^\infty$ isomorphism,
$$
T^{0,1}X\, \stackrel{g_1}{\longrightarrow}\,(T^{1,0}X)^*\, ,
$$
where $T^{\mathbb R}X\otimes {\mathbb C}\,=\, T^{1,0}X\oplus T^{0,1}X$ is the type decomposition with
respect to the almost complex structure $J_1$; also $J_2$ produces a $C^\infty$ isomorphism
$$
T^{1,0}X\, \stackrel{J'_2}{\longrightarrow}\,T^{0,1}X\, .
$$
The composition of homomorphisms
$$
T^{1,0}X\, \stackrel{J'_2}{\longrightarrow}\,T^{0,1}X \, \stackrel{g_1}{\longrightarrow}\,(T^{1,0}X)^*\, ,
$$
which is a section of $(T^{1,0}X)^*\otimes (T^{1,0}X)^*$, is actually is a holomorphic symplectic form on the
compact K\"ahler manifold $(X,\, J_1,\, g)$. The compact K\"ahler manifold $(X,\, J_1,\, g)$ is Ricci--flat.
Conversely, if a compact K\"ahler manifold admits a
holomorphic symplectic form, then its canonical line bundle is holomorphically trivial and hence it admits
a Ricci--flat K\"ahler metric \cite{Ya}. Let $(X,\, J_1 ,\, g)$ be a Ricci--flat compact K\"ahler
manifold equipped with a holomorphic symplectic form. Then we may recover $J_2$ by reversing the above
construction. Finally, we have $J_3\,=\, J_1\circ J_2$.

Let $X$ be a compact K\"ahler Ricci--flat manifold admitting a holomorphic
symplectic form. Fix a Ricci--flat K\"ahler form $\omega$ on $X$ (such
a K\"ahler form exists \cite{Ya}), and take a holomorphic symplectic
form $\Phi$ on $X$. It is known that $\Phi$ is parallel (meaning, covariant
constant) with respect to the Levi--Civita connection on $X$
associated to $\omega$ \cite[p.~760, ``Principe~de~Bochner'']{Be},
\cite[p.~142]{YB}.

Let $M$ denote the moduli space of compact complex submanifolds of $X$
that are Lagrangian with respect to the symplectic form $\Phi$. Consider the
corresponding universal family of Lagrangians
\begin{equation}\label{ez}
{\mathcal Z}\, \longrightarrow\, M\, .
\end{equation}
Let
$$
\varpi\, :\, \widehat{\mathcal{A}}\, \longrightarrow\, M
$$
be the relative Albanese over $M$. So for any Lagrangian $L\, \in\, M$,
the fiber of $\widehat{\mathcal{A}}$ over $L$ is $\text{Alb}(L)\,=\,
H^0(L,\,\Omega^1_L)^*/H_1\,(L,\,\mathbb Z)$. This $\varphi$ is a holomorphic family
of compact complex tori over $M$.

We prove the following (see Theorem \ref{theo1}):

\textit{The complex manifold $\widehat{\mathcal{A}}$ has a natural holomorphic
symplectic form.}

The symplectic form on $\widehat{\mathcal{A}}$ is constructed using the
canonical Liouville symplectic form on the holomorphic cotangent bundle
$T^*M$ of $M$. The symplectic form $\Phi$ on $X$ is implicitly used in the
construction of the symplectic form on $\widehat{\mathcal{A}}$. Recall that the
Lagrangian submanifolds, and hence $M$, are defined using $\Phi$.

We prove the following (see Lemma \ref{lem-1}):

\textit{The projection $\varpi\, :\, \widehat{\mathcal{A}}\, \longrightarrow\, M$ defines a
completely integrable structure on the symplectic
manifold $\widehat{\mathcal{A}}$. In particular, the
fibers of $\varpi$ are Lagrangians with respect to the
symplectic form on $\widehat{\mathcal{A}}$.}

In Section \ref{se4} we consider the relative Picard bundle over $M$ for the
family ${\mathcal Z}$ in \eqref{ez}. Let
$$
\varpi_0\, :\, \mathcal{A}\, \longrightarrow\, M
$$
be the relative Picard bundle for the family ${\mathcal Z}$.

We prove that $\mathcal{A}$ is equipped with a natural holomorphic symplectic structure;
see Proposition \ref{proposi1}.

Let $\Theta_{\mathcal A}$ denote the above mentioned holomorphic symplectic structure
on $\mathcal{A}$. The following lemma is proved (see Lemma \ref{lem2}):

\textit{The projection $\varpi_0\,:\, \mathcal{A}\, \longrightarrow\, M$ defines a
completely integrable structure on $\mathcal{A}$ for the symplectic
form $\Theta_{\mathcal A}$.}

These results are natural generalizations of some known cases of integrable systems, such as the
Hitchin system \cite{Hi1} or the Mukai system \cite{Mu} (see also \cite{DEL}).
One of our motivations has been mirror symmetry; we hope to come back to the study of
$\mathcal{A}$ and $\widehat{\mathcal{A}}$ from the point of view of
hyperK\"ahler geometry.

\section{Cotangent bundle of family of Lagrangians}\label{sect2}

Let $X$ be a compact K\"ahler Ricci--flat manifold of complex dimension $2d$ equipped with a K\"ahler
form $\omega$. Let $\Phi$ be a holomorphic symplectic form on $X$ which is parallel
with respect to the Levi--Civita connection on $X$ given by the K\"ahler metric on $X$
associated to $\omega$.

A complex Lagrangian submanifold of $X$ is a compact complex submanifold $L\,\subset\, X$
of complex dimension $d$ such that $\iota^*\Phi\,=\,0$, where
\begin{equation}\label{e1}
\iota\, :\, L\, \hookrightarrow\, X
\end{equation}
is the inclusion map.

It is known that the infinitesimal deformations of a complex Lagrangian submanifold
of $X$ are unobstructed \cite{Mc}, \cite{Vo}, \cite{Hi2}. Furthermore, the moduli space
of complex Lagrangian submanifolds of $X$ is a special K\"ahler manifold \cite[p. 84,
Theorem 3]{Hi2}.

Let $M$ be the moduli space of complex Lagrangian submanifolds of $X$. Let
$$
L\,\subset\, X
$$
be a complex Lagrangian submanifold. The point of $M$ representing $L$ will also be
denoted by $L$. Let $N_L\, \longrightarrow\, L$ be the normal bundle
of $L\, \subset\, X$; it is a quotient bundle of $\iota^*TX$ of rank $d$, where $\iota$ is
the map in \eqref{e1}.
The infinitesimal deformations of the complex submanifold
$L$ are parametrized by $H^0(L,\, N_L)$. Since
$L$ is complex Lagrangian, the holomorphic symplectic form $\Phi$ on $X$ produces a holomorphic isomorphism
$$
N_L\,\stackrel{\sim}{\longrightarrow}\, (TL)^*\,=\, \Omega^1_L\, ,
$$
where $TL$ (respectively, $\Omega^1_L$) is the holomorphic tangent (respectively, cotangent)
bundle of $L$. Using this isomorphism we have
$$
H^0(L,\, N_L)\,=\, H^0(L,\, \Omega^1_L)\, .
$$
Among the infinitesimal deformations of the complex submanifold
$L$, there are those which arise from deformations within the category of complex Lagrangian submanifolds,
meaning those arise from deformations of complex Lagrangian submanifolds as Lagrangian submanifolds.
An infinitesimal deformation $$\alpha\, \in\, H^0(L,\, N_L)\,=\, H^0(L,\, \Omega^1_L)$$ of $L$ lies
in this subclass if and only if the holomorphic $1$--form $\alpha$
on $L$ is closed \cite[pp.~78--79]{Hi2}. But any holomorphic $1$--form on $L$ is closed because $L$ is K\"ahler.
Therefore, $M$ is in fact an open subset of the corresponding Douady space for $X$.

Consider the Ricci--flat K\"ahler form $\omega$ on $X$. The form
\begin{equation}\label{wl}
\omega_L\, :=\, \iota^*\omega
\end{equation}
on $L$ is also K\"ahler, where $\iota$ is the map in \eqref{e1}.
Therefore, the pairing
\begin{equation}\label{e2}
\phi_L\, :\, H^0(L,\, \Omega^1_L)\otimes H^1(L,\, {\mathcal O}_L)\, \longrightarrow\,
\mathbb C\, , \ \ w\otimes c\, \longmapsto\, \int_L w\wedge c\wedge\omega^{d-1}_L
\end{equation}
is nondegenerate. We shall identify $H^1(L,\, {\mathcal O}_L)$ with $H^0(L,\, \Omega^1_L)^*$
using this nondegenerate pairing.

It was noted above that
\begin{equation}\label{n1}
T_LM\,=\, H^0(L,\, N_L)\,=\, H^0(L,\, \Omega^1_L)\, .
\end{equation}
Using the pairing in \eqref{e2} and \eqref{n1}, we
have
\begin{equation}\label{e3}
T^*_LM\,=\, H^0(L,\, \Omega^1_L)^*\,=\, H^1(L,\, {\mathcal O}_L)\, .
\end{equation}

Let
\begin{equation}\label{e-1}
{\mathcal Z}\, \subset\, X\times M
\end{equation}
be the universal family of complex Lagrangians over $M$. So ${\mathcal Z}$ is the locus of
all $(x,\, L')\, \in\, X\times M$ such that $x\, \in\, L'
\, \subset\, X$. Consider the natural
projections $p_X\, :\, X\times M\, \longrightarrow\, X$ and $p_M\, :\, X\times M\, \longrightarrow\, M$. Let
\begin{equation}\label{pq}
p\, :\, {\mathcal Z}\, \longrightarrow\, X \ \ \text{ and }\ \
q\, :\, {\mathcal Z}\, \longrightarrow\, M
\end{equation}
be the restrictions of $p_X$ and $p_M$ respectively to the submanifold ${\mathcal Z}\, \subset\, X\times M$.
So $p(q^{-1}(L'))\, \subset\, X$ for every $L'\, \in\, M$ is
the Lagrangian $L'$ itself.

Let $\Omega^1_{{\mathcal Z}/M}\, \longrightarrow\, {\mathcal Z}$ be the relative cotangent bundle
for the projection $q$ to $M$ in \eqref{pq}. It fits in the short exact sequence of holomorphic vector bundles
$$
0 \, \longrightarrow\, q^*\Omega^1_M \, \longrightarrow\, \Omega^1_{\mathcal Z}
\, \longrightarrow\, \Omega^1_{{\mathcal Z}/M}\, \longrightarrow\, 0
$$
over $M$.
The direct image $R^1q_*{\mathcal O}_{\mathcal Z}$ fits in the short exact sequence of sheaves on $M$
\begin{equation}\label{a1}
0\, \longrightarrow\,R^0q_*\Omega^1_{{\mathcal Z}/M}\, \longrightarrow\, R^1q_*\underline{\mathbb C}
\, \longrightarrow\, R^1q_*{\mathcal O}_{\mathcal Z}\, \longrightarrow\, 0\, ,
\end{equation}
where $\underline{\mathbb C}$ is the constant sheaf on ${\mathcal Z}$ with stalk $\mathbb C$. The direct image
$R^1q_*\underline{\mathbb C}$ is a flat complex vector bundle, equipped with the Gauss--Manin connection.
We briefly recall the construction of the Gauss--Manin connection. For any point $L'\,=\, y \, \in\, M$, let
$U_y\, \subset\, M$ be a contractible open neighborhood of $y$. Since $U_y$ is contractible,
the inverse image $q^{-1}(U_y)$ is diffeomorphic to $U_y\times L'$ such that the diffeomorphism
between $q^{-1}(U_y)$ and $U_y\times L'$ takes $q$ to the natural projection from $U_y\times L'$ to $U_y$.
Using this diffeomorphism, the restriction of $R^1q_*\underline{\mathbb C}$ to $U_y$ coincides with the
trivial vector bundle
\begin{equation}\label{tvb}
U_y\times H^1(L',\, {\mathbb C}) \, \longrightarrow\, U_y
\end{equation}
with fiber $H^1(L',\, {\mathbb C})$. Using this isomorphism between $(R^1q_*\underline{\mathbb C})\vert_{U_y}$
and the trivial vector bundle $U_y\times H^1(L',\, {\mathbb C})$ in \eqref{tvb}, the trivial connection on
the trivial vector bundle in \eqref{tvb} produces a flat connection on $(R^1q_*\underline{\mathbb C})\vert_{U_y}$.
This connection on $(R^1q_*\underline{\mathbb C})\vert_{U_y}$ does not depend on the choice of the
diffeomorphism between $q^{-1}(U_y)$ and $U_y\times L'$. Consequently, these locally defined flat connections
on $R^1q_*\underline{\mathbb C}$ patch together compatibly to define a flat connection on
$R^1q_*\underline{\mathbb C}$. This flat connection is the Gauss--Manin connection mentioned above.

Since the Gauss--Manin connection on
$R^1q_*\underline{\mathbb C}$ is flat, and $M$ is a complex manifold, the Gauss--Manin connection
produces a natural holomorphic structure
on the $C^\infty$ vector bundle $R^1q_*\underline{\mathbb C}$. The direct image
$R^0q_*\Omega^1_{{\mathcal Z}/M}$ is a holomorphic subbundle of $R^1q_*\underline{\mathbb C}$,
but it is not preserved by the flat connection in general. The holomorphic structure
on the quotient $R^1q_*{\mathcal O}_{\mathcal Z}$ induced by that of $R^1q_*\underline{\mathbb C}$ coincides with
its own holomorphic structure; the fiber of $R^1q_*{\mathcal O}_{\mathcal Z}$
over any $L'\, \in\, M$ is $H^1(L',\, {\mathcal O}_{L'})$.

Since $\omega_L$ in \eqref{wl} is the restriction of a global K\"ahler form on $X$, the section of
$R^2q_*\underline{\mathbb C}$
$$
M\,\longrightarrow\, R^2q_*\underline{\mathbb C}\, ,\ \ L' \,\longmapsto\, [\omega_{L'}]\,=\,
[\omega\vert_{L'}]\,\in\, H^2(L', \, {\mathbb C})
$$
is covariant constant with respect to the Gauss--Manin connection on $R^2q_*\underline{\mathbb C}$.
Consequently, the homomorphism
\begin{equation}\label{ep}
(R^1 q_*\underline{\mathbb C})\otimes (R^1 q_*\underline{\mathbb C})\, \longrightarrow\,
\underline{\mathbb C}\, ,
\end{equation}
that sends any $v\otimes w \, \in\,
(R^1 q_*\underline{\mathbb C})_t\otimes (R^1 q_*\underline{\mathbb C})_t$, $t\, \in\, M$,
to
$$
\int_{q^{-1}(t)} v\wedge w\wedge (\omega\vert_{q^{-1}(t)})^{d-1}\, \in\, \mathbb C
$$
is also covariant constant with respect to the connection on $(R^1 q_*\underline{\mathbb C})
\otimes (R^1 q_*\underline{\mathbb C})$ induced by the Gauss--Manin connection
on $R^1 q_*\underline{\mathbb C}$. Hence the pairing
in \eqref{ep} produces a holomorphic isomorphism of vector bundles
\begin{equation}\label{ep1}
(q_*\Omega^1_{{\mathcal Z}/M})^* \,\stackrel{\sim}{\longrightarrow} \, R^1q_*{\mathcal O}_{\mathcal Z}
\end{equation}
on $M$. The restriction of this isomorphism to any point $L\, \in\, M$ coincides
with the isomorphism $H^0(L,\, \Omega^1_L)^*\,=\, H^1(L,\, {\mathcal O}_L)$ in \eqref{e3}.

On the other hand, the pointwise isomorphisms in \eqref{n1} combine together
to produce a holomorphic isomorphism of vector bundles
\begin{equation}\label{ep2}
\Omega^1_M \,\stackrel{\sim}{\longrightarrow} \, (q_*\Omega^1_{{\mathcal Z}/M})^*
\end{equation}
on $M$. Composing the isomorphisms in \eqref{ep1} and \eqref{ep2}, we obtain
a holomorphic isomorphism of vector bundles
\begin{equation}\label{e4}
\chi\, :\, \Omega^1_M \,\longrightarrow \, R^1q_*{\mathcal O}_{\mathcal Z}
\end{equation}
over $M$.

For notational convenience, the total space of $R^1q_*{\mathcal O}_{\mathcal Z}$ will be
denoted by ${\mathcal Y}$. Let
\begin{equation}\label{e6}
\gamma\, :\, {\mathcal Y}\,\longrightarrow \, M
\end{equation}
be the natural projection. Consider the canonical Liouville $1$-form on $\Omega^1_M$.
Using the isomorphism $\chi$ in \eqref{e4} this Liouville $1$-form on $\Omega^1_M$ gives
a holomorphic $1$-form on ${\mathcal Y}$. Let
\begin{equation}\label{e5}
\theta\,\in\, H^0({\mathcal Y}, \, \Omega^1_{\mathcal Y})
\end{equation}
be the holomorphic $1$-form given by the Liouville $1$-form on $\Omega^1_M$. Note that
for any $L\,\in\, M$, and any $v\,\in\, \gamma^{-1}(L)$, the form
$$
\theta(v)\, :\, T_v{\mathcal Y}\,\longrightarrow \, \mathbb C
$$
coincides with the composition of homomorphisms
$$
T_v{\mathcal Y}\,\stackrel{d\gamma}{\longrightarrow}\, T_L M\,=\, H^0(L,\, \Omega^1_L)
\,\stackrel{\phi_L(-,v)}{\longrightarrow}\,\mathbb C\, ,
$$
where $\phi_L$ is the bilinear pairing constructed in \eqref{e2}, and $d\gamma$ is the
differential of the projection $\gamma$ in \eqref{e6}; the above identification $$T_L M\,=\,
H^0(L,\, \Omega^1_L)$$ is the one constructed in \eqref{n1}.

It is straight-forward to check that the $2$-form
\begin{equation}\label{e10}
d\theta\, \in\, H^0({\mathcal Y},\, \Omega^2_{\mathcal Y})
\end{equation}
is a holomorphic symplectic form on the manifold
${\mathcal Y}$ in \eqref{e6}. Indeed, $d\theta$ evidently coincides with
the $2$-form on ${\mathcal Y}$ given by the Liouville symplectic form on
$\Omega^1_M$ via the isomorphism $\chi$ in \eqref{e4}.

\section{The family of Albanese tori}

Take a compact complex Lagrangian submanifold $L\subset X$ represented by a point of $M$.
We know that $T_L^*M\,\cong\, H^0(L,\,\Omega^1_L)^*$
(see \eqref{n1}). Note that the non-degenerate pairing
$$
H^{1,0}(L)\otimes H^{d-1,d}(L)\, \longrightarrow\,
\mathbb C\, , \ \ \alpha\otimes \beta\, \longmapsto\, \int_L \alpha\wedge \beta\, ,
$$
which is also the Serre duality pairing, yields an isomorphism
$$H^0(L,\,\Omega^1_L)^*\,\cong\, H^d(L,\,\Omega^{d-1}_L)\, .$$

For a fixed $L$, we have the Hodge decomposition $$H^{2d-1}(L,\,\mathbb C)\,=\,H^{d,d-1}(L)\oplus H^{d-1,d}(L)\, ;$$
but if we move $L$ in the family $M$, meaning if we consider the universal family $$q\,:\,\mathcal{Z}
\, \longrightarrow\, M$$ in \eqref{e-1},
then only $R^{d-1}q_*\Omega^d_{{\mathcal Z}/M}$ is a holomorphic subbundle of $R^{2d-1}q_*\underline{\mathbb C}$,
and we have the short exact sequence of holomorphic vector bundles
\begin{equation}\label{n2}
0\, \longrightarrow\,R^{d-1}q_*\Omega^d_{\mathcal Z/M}\, \longrightarrow\, R^{2d-1}q_*\underline{\mathbb C}
\, \longrightarrow\, R^dq_*\Omega^{d-1}_{\mathcal Z/M}\, \longrightarrow\, 0\, ,
\end{equation}
on $M$.

The holomorphic vector bundle $R^{2d-1}q_*\underline{\mathbb C}$ is equipped with the Gauss--Manin connection,
which is an integrable connection. The quotient $R^dq_*\Omega^{d-1}_{\mathcal Z/M}$, in \eqref{n2},
of $R^{2d-1}q_*\underline{\mathbb C}$ is a holomorphic vector
bundle on $M$ with fiber $H^d(L',\,\Omega^{d-1}_{L'})$ over any $L'\, \in\, M$.

The homomorphism
\begin{equation}\label{ep4}
(R^1 q_*\underline{\mathbb C})\otimes (R^{2d-1} q_*\underline{\mathbb C})\, \longrightarrow\,
\underline{\mathbb C}\, ,
\end{equation}
that sends any $\alpha\otimes \beta \, \in\,
(R^1 q_*\underline{\mathbb C})_t\otimes (R^{2d-1} q_*\underline{\mathbb C})_t$, $t\, \in\, M$,
to
$$
\int_{q^{-1}(t)} v\wedge w \in\, \mathbb C
$$
is covariant constant with respect to the connection on $(R^1 q_*\underline{\mathbb C})\otimes
(R^{2d-1} q_*\underline{\mathbb C})$ induced by the Gauss--Manin connections on
$R^1 q_*\underline{\mathbb C}$ and $R^{2d-1} q_*\underline{\mathbb C}$. Consequently,
the pairing in \eqref{ep4} yields a holomorphic isomorphism of vector bundles
$$
(q_*\Omega^1_{{\mathcal Z}/M})^* \,\stackrel{\sim}{\longrightarrow} \, R^dq_*\Omega^{d-1}_{\mathcal Z/M}
$$
over $M$. Combining this isomorphism with the isomorphism in \eqref{ep2}
we get a holomorphic isomorphism of vector bundles
\begin{equation}\label{e4'}
\widehat{\chi}\, :\, \Omega^1_M \,\stackrel{\sim}{\longrightarrow} \, R^dq_*\Omega^{d-1}_{\mathcal Z/M}
\end{equation}
over $M$.

We shall denote the total space of the holomorphic vector bundle
$R^dq_*\Omega^{d-1}_{\mathcal Z/M}$ by $\mathcal{W}$, so
\begin{equation}\label{e11}
\mathcal{W}\, :=\, R^dq_*\Omega^{d-1}_{\mathcal Z/M}\, \stackrel{\widehat\gamma}{\longrightarrow}\, M
\end{equation}
is a holomorphic fiber bundle.

As in Section \ref{sect2}, consider the canonical Liouville
holomorphic $1$-form on the total space of $\Omega^1_M$. Using the isomorphism in \eqref{e4'},
this Liouville $1$-form on the total space of $\Omega^1_M$ produces a
holomorphic $1$-form on $\mathcal{W}$ in \eqref{e11}. Let
$$
\theta'\, \in\, H^0(\mathcal{W}, \, \Omega^1_{\mathcal{W}})
$$
be this holomorphic $1$-form on $\mathcal{W}$. We note that
\begin{equation}\label{e8}
d\theta'\, \in\, H^0(\mathcal{W}, \, \Omega^2_{\mathcal{W}})
\end{equation}
is a holomorphic symplectic form on $\mathcal{W}$. Indeed, the isomorphism in
\eqref{e4'} takes $d\theta'$ to the Liouville symplectic form on the
total space of $\Omega^1_M$.

\begin{remark}\label{rem0}
Note that while the K\"ahler form $\omega$ on $X$ was used in the construction of the
symplectic form $d\theta$ on $\mathcal Y$ in \eqref{e10} (see the pairing in \eqref{ep}), the construction of
$d\theta'$ in \eqref{e8} does not use the K\"ahler form $\omega$ on $X$. We recall that the isomorphism
in \eqref{ep2} is constructed from the the pointwise isomorphisms in \eqref{n1}. Note that the
isomorphism in \eqref{n1} does not depend on the K\"ahler form $\omega$.
\end{remark}

The Albanese $\text{Alb}(Y)$ of a compact K\"ahler manifold $Y$ is defined to be
$$
\text{Alb}(Y)\,=\, H^0(Y,\,\Omega^1_Y)^*/H_1\,(Y,\,\mathbb Z)\,=\, H^n(Y,\,\Omega^{n-1}_Y)/H_1\,(Y,\,\mathbb Z)\, ,
$$
where $n\,=\, \dim_{\mathbb C} Y$ (see \cite[p.~331]{GH}). It is a compact
complex torus.

For each point $L\,\in\, M$, consider the composition of homomorphisms
$$
H^{2d-1}(L,\,\mathbb Z)\,\longrightarrow\, H^{2d-1}(L,\,{\mathbb C})\,\longrightarrow\,
H^{2d-1}(L,\,{\mathbb C})/H^{d-1}(L,\,\Omega_L^d)\,=\, H^d(L,\,\Omega_L^{d-1})
$$
(see \eqref{n2}). It produces a homomorphism
\begin{equation}\label{e14}
R^{2d-1}q_*\underline{\mathbb Z}\, \longrightarrow\,\mathcal{W}\, ,
\end{equation}
where $\mathcal{W}$ is defined in \eqref{e11}.

\begin{remark}\label{r-1}
The Gauss--Manin connection on $R^{2d-1}q_*\underline{\mathbb C}\,\longrightarrow\, M$ evidently preserves
the subbundle of lattices $$R^{2d-1}q_*\underline{\mathbb Z}\,\subset\,R^{2d-1}q_*\underline{\mathbb C}\, .$$
From this it follows immediately that the $C^\infty$ submanifold $R^{2d-1}q_*\underline{\mathbb Z}\,\subset\,
{\mathcal W}$ in \eqref{e14} is in fact a complex submanifold.
\end{remark}

The quotient
\begin{equation}\label{e9}
\widehat{\mathcal{A}}\,:=\,\mathcal{W}/(R^{2d-1}q_*\underline{\mathbb Z})\,\longrightarrow\, M
\end{equation}
for the homomorphism in \eqref{e14}
is in fact a holomorphic family of compact complex tori over $M$. Note that the fiber
of $\widehat{\mathcal{A}}$ over each $L\,\in\, M$
is the Albanese torus $$\mathrm{Alb}(L)\,=\,H^d(L,\,\Omega_L^{d-1})/H^{2d-1}(L,\,\mathbb Z)\, .$$

For any complex Lagrangian $L\, \in\, M$, using Serre duality,
$$
H^d(L,\,\Omega_L^{d-1})\,=\, H^0(L,\,\Omega_L^1)^*\, ,
$$
and the underlying real vector space for $H^0(L,\,\Omega_L^1)^*$ is identified with
$$H^1(L,\, {\mathbb R})^*\,=\, H_1(L,\, {\mathbb R})\, .$$ Using Poincar\'e duality for $L$, we have
$H^{2d-1}(L,\,\mathbb Z)/\text{Torsion}\,=\, H_1(L,\,\mathbb Z)/\text{Torsion}$. Consequently,
$\widehat{\mathcal{A}}$ in \eqref{e9} admits the following isomorphism:
\begin{equation}\label{e12}
\widehat{\mathcal{A}}\, =\,\mathcal{W}/(R^{2d-1}q_*\underline{\mathbb Z})\,=\,
\mathcal{W}/\widetilde{H}_1(\mathbb Z)\,=\,
(q_*\Omega^1_{\mathcal Z/M})^*/\widetilde{H}_1(\mathbb Z)\, ,
\end{equation}
where $\widetilde{H}_1(\mathbb Z)$ is the local system on $M$ whose stalk over any $L\, \in\, M$ is
$H_1(L,\,\mathbb Z)/\text{Torsion}$. Also we have the isomorphism of real tori
\begin{equation}\label{e21}
\widehat{\mathcal{A}}\, =\, (R^1q_*\underline{\mathbb R})^*/\widetilde{H}_1(\mathbb Z)\,=\,
\widetilde{H}_1({\mathbb R})/\widetilde{H}_1(\mathbb Z)\, ,
\end{equation}
where $\widetilde{H}_1({\mathbb R})$ is the local system on $M$ whose stalk over any $L\, \in\, M$ is
$H_1(L,\,{\mathbb R})$.

\begin{theorem}\label{theo1}
The $2$-form $d\theta'$ in \eqref{e8} on ${\mathcal W}$ descends to the quotient torus
$\widehat{\mathcal A}$ in \eqref{e9}.
\end{theorem}

\begin{proof}
Take a point $L_0\,\in\, M$. In \cite{Hi2} Hitchin constructed a $C^\infty$ coordinate function
on $M$ defined around the point $L_0\,\in\, M$
that takes values in $H^1(L_0,\, {\mathbb R})$ \cite[p.~79, Theorem 2]{Hi2}; we will briefly recall
this construction.

Take any
\begin{equation}\label{ec}
c\,\in\, H_1(L_0,\, {\mathbb Z})/{\rm Torsion}\, .
\end{equation}
Let $U\,\subset\, M$
be a contractible neighborhood of the point $L_0$. Choose a $S^1$-subbundle of
the fiber bundle $\mathcal Z$ (see \eqref{e-1}) over $U$
\begin{equation}\label{fb}
B\,\stackrel{\iota_U}{\hookrightarrow}\, {\mathcal Z}\vert_U \, \stackrel{q}{\longrightarrow}\, U
\end{equation}
such that the fiber of the $S^1$-bundle $B$ over $L_0$ represents the homology
class $c$ in \eqref{ec}; recall that the fiber of $\mathcal Z$ over the point $L_0\,\in\,
M$ is the Lagrangian $L_0$ itself.

Consider the symplectic form $\Phi$ on $X$. Integrating $\iota^*_Up^*
{\rm Re}(\Phi)$ along the fibers of $B$, where $\iota_U$ and $p$ are the maps in \eqref{fb}
and \eqref{pq} respectively, we get a closed $1$-form $\xi_c$ on $U$. Let $f_c$ be the
unique function on $U$ such that $f_c(L_0)\,=\, 0$ and $df_c\,=\, \xi_c$. Now, let
$$
\mu\, :\, U \, \longrightarrow\, H^1(L_0,\, {\mathbb R})
$$
be the function uniquely determined by the condition that $\phi_{L_x}(c\otimes \mu(x)) \,=\, f_c(x)$ for all
$x\,\in\, U$ and $c\,\in\, H_1(L_0,\, {\mathbb Z})/{\rm Torsion}$, where
$\phi_{L_x}$ is the pairing constructed as in \eqref{e2} for the complex Lagrangian $L_x\,=\, q^{-1}(x)
\, \subset\, X$, where $q$ is the projection
in \eqref{pq}. This $\mu$ is a local diffeomorphism \cite[p.~79, Theorem 2]{Hi2}.

Using the K\"ahler form $\omega_{L_0}\,:=\, \omega\vert_{L_0}$ on $L_0$ (see \eqref{wl}), we identify
$H^1(L_0,\, {\mathbb R})$ with $H_1(L_0,\, {\mathbb R})$ as follows. Since the pairing
$$
H^1(L_0,\, {\mathbb R})\otimes H^1(L_0,\, {\mathbb R})\,\longrightarrow {\mathbb R}\, , \ \
v\otimes w\, \longmapsto\, \int_L v\wedge w\wedge\omega^{d-1}_{L_0}
$$
is nondegenerate, it produces an isomorphism
\begin{equation}\label{ei}
H^1(L_0,\, {\mathbb R})\,\stackrel{\sim}{\longrightarrow}\,
H^1(L_0,\, {\mathbb R})^* \,=\, H_1(L_0,\, {\mathbb R})\, .
\end{equation}
On the other hand, there is the natural homomorphism $H^1(L_0,\, {\mathbb Z})\, \longrightarrow\,
H^1(L_0,\, {\mathbb R})$. Let
\begin{equation}\label{ga}
\Gamma\, \subset\, H_1(L_0,\, {\mathbb R})
\end{equation}
be the subgroup that corresponds to
$H^1(L_0,\, {\mathbb Z})$ by the isomorphism in \eqref{ei}.

Using the above coordinate function $\mu$ on $U$, we have
$$
{\mathcal T}^*U\,\stackrel{\sim}{\longrightarrow}\, {\mathcal T}^*\mu(U) \,=\,\mu(U)\times
H^1(L_0,\, {\mathbb R})^*\,=\, \mu(U)\times H_1(L_0,\, {\mathbb R})\, ,
$$
where ${\mathcal T}^*$ denotes the real cotangent bundle.

The Liouville symplectic form on ${\mathcal T}^*\mu(U)$ is clearly the constant $2$-form
on $$H^1(L_0,\, {\mathbb R})\times H_1(L_0,\, {\mathbb R})$$ given by the
natural isomorphism of $H_1(L_0,\, {\mathbb R})$ with $H^1(L_0,\, {\mathbb R})^*$.
From this it follows immediately that
$$
\mu(U)\times \Gamma\, \subset\, \mu(U)\times H_1(L_0,\, {\mathbb R})
$$
is a Lagrangian submanifold with respect to the Liouville symplectic form on ${\mathcal T}^*\mu(U)$,
where $\Gamma$ defined in \eqref{ga}.

Since $\mu(U)\times \Gamma\, \subset\, \mu(U)\times H_1(L_0,\, {\mathbb R})$ is a Lagrangian submanifold,
it follows that for ${\mathcal W}\,=\,
(R^1q_*\underline{\mathbb R})^*$ (see \eqref{e21} and \eqref{e11}), the image of the natural map
$$
\widetilde{H}_1(\mathbb Z) \, \longrightarrow\, (R^1q_*\underline{\mathbb R})^*\,=\, {\mathcal W}
$$
(see \eqref{e12}) is Lagrangian with respect to the real symplectic form $\text{Re}(d\theta')$ on
$\mathcal W$, where $d\theta'$ is constructed in \eqref{e8}.

It was noted in Remark \ref{r-1} that $R^{2d-1}q_*\underline{\mathbb Z}$ is a complex submanifold of ${\mathcal W}$.
Consequently, the $2$-form on $R^{2d-1}q_*\underline{\mathbb Z}$ obtained by restricting the
holomorphic $2$-form $d\theta'$ on $\mathcal W$ is also holomorphic. Since the real part of the holomorphic
$2$-from on $R^{2d-1}q_*\underline{\mathbb Z}$ given by $d\theta'$ vanishes identically, we conclude that the
holomorphic $2$-from on $R^{2d-1}q_*\underline{\mathbb Z}$, given by $d\theta'$, itself vanishes
identically. Therefore, $R^{2d-1}q_*\underline{\mathbb Z}$ is a Lagrangian submanifold of the holomorphic symplectic
manifold ${\mathcal W}$ equipped with the holomorphic symplectic form $d\theta'$.

To complete the proof we recall a general property of the Liouville symplectic form.

Let $N$ be a manifold and $\alpha$ a $1$-form on $N$. Let
$$
t\, :\, T^*N \,\longrightarrow\, T^*N
$$
be the diffeomorphism that sends any $v \,\in\, T^*_nN$ to $v+\alpha(n)$. If $\psi$ is the
Liouville symplectic form on $T^*N$, the
$$
t^*\psi \,=\, \psi+d\alpha \, .
$$
In particular, the map $t$ preserves $\psi$ if and only if the form $\alpha$ is closed.
Also, the image of $t$ is Lagrangian submanifold of $T^*N$ for $\psi$ if and only
$\alpha$ is closed.

Since $$R^{2d-1}q_*\underline{\mathbb Z}$$ is a Lagrangian submanifold of the symplectic manifold
$({\mathcal W},\, d\theta')$, from the above property of the Liouville symplectic form it
follows immediately that the $2$-form $d\theta'$ on ${\mathcal W}$ descends to the quotient space
$\widehat{\mathcal A}$ in \eqref{e9}.
\end{proof}

Let
\begin{equation}\label{qa}
q_{\widehat{\mathcal A}}\, :\, {\mathcal W}\, \longrightarrow\, \widehat{\mathcal A}\, :=
\,{\mathcal W}/(R^{2d-1}q_*\underline{\mathbb Z})
\end{equation}
be the quotient map (see \eqref{e9}).
From Theorem \ref{theo1} we know that there is a unique $2$-form
\begin{equation}\label{ta}
\Theta_{\widehat{\mathcal A}}\, \in\, H^0(\widehat{\mathcal A},\, \Omega^2_{\widehat{\mathcal A}})
\end{equation}
such that
\begin{equation}\label{ta2}
q^*_{\widehat{\mathcal A}}\Theta_{\widehat{\mathcal A}}\,=\, d\theta'\, ,
\end{equation}
where $q_{\mathcal A}$ is the map
in \eqref{qa}. Since $d\theta'$ is a holomorphic symplectic form, it follows immediately that
$\Theta_{\widehat{\mathcal A}}$ is a holomorphic symplectic form on $\widehat{\mathcal A}$.

The projection $\widehat{\gamma}\, :\, {\mathcal W}\,\longrightarrow \, M$ in \eqref{e11} clearly
descends to a map from ${\mathcal A}$ to $M$. Let
\begin{equation}\label{vp}
\varpi\, :\, \widehat{\mathcal{A}}\, \longrightarrow\, M
\end{equation}
be the map given by $\widehat{\gamma}$; so we have
$$
\widehat{\gamma}\,=\, \varpi\circ q_{\widehat{\mathcal A}}\, ,
$$
where $q_{\widehat{\mathcal A}}$ is constructed in \eqref{qa}.

\begin{lemma}\label{lem-1}
The projection $\varpi$ in \eqref{vp} defines a
completely integrable structure on $\widehat{\mathcal{A}}$ for the symplectic
form $\Theta_{\widehat{\mathcal A}}$ constructed in \eqref{ta}. In particular, the
fibers of $\varpi$ are Lagrangians with respect to the
symplectic form $\Theta_{\widehat{\mathcal A}}$.
\end{lemma}

\begin{proof}
Recall that $\mathcal W$ is holomorphically identified with $\Omega^1_M$ by the map
$\widehat{\chi}$ in \eqref{e4'} (see \eqref{e11}).
This map $\widehat\chi$ takes the Liouville symplectic form on $\Omega^1_M$ to the
symplectic form $d\theta'$ on $\mathcal W$. Therefore, from \eqref{qa} and \eqref{ta2} we conclude that
$\widehat{\mathcal A}$ is locally isomorphic to $\Omega^1_M$ such that the projection $\varpi$ is taken to the
natural projection $\Omega^1_M\, \longrightarrow\, M$, and the symplectic form $\Theta_{\widehat{\mathcal A}}$
on $\widehat{\mathcal A}$ is taken to the Liouville symplectic form on $\Omega^1_M$. The lemma follows
immediately from these, because the natural projection $\Omega^1_M\, \longrightarrow\, M$ defines a
completely integrable structure on $\Omega^1_M$ for the Liouville symplectic form.
\end{proof}

\section{The relative Picard group}\label{se4}

For any $L\,\in\, M$, consider the homomorphisms
\begin{equation}\label{h1}
H^1(L,\, {\mathbb Z}) \,\longrightarrow \, H^1(L,\, {\mathbb C})\,\longrightarrow \, H^1(L,
\, {\mathcal O}_L)\, ,
\end{equation}
where $H^1(L,\, {\mathbb Z}) \,\longrightarrow \, H^1(L,\, {\mathbb C})$ is the natural
homomorphism given by the inclusion of $\mathbb Z$ in $\mathbb C$, and the projection
$H^1(L,\, {\mathbb C})\,\longrightarrow \, H^1(L,\, {\mathcal O}_L)$ corresponds to
the isomorphism $$H^1(L,\, {\mathbb C})/H^0(L,\, \Omega^1_L)\,=\,
H^1(L,\, {\mathcal O}_L)$$ (see \eqref{a1}).
The image of the composition of homomorphisms in \eqref{h1} is actually a cocompact lattice in $H^1(L,
\, {\mathcal O}_L)$; so $H^1(L,\, {\mathcal O}_L)/H^1(L,\, {\mathbb Z})$ is a compact complex torus.
We note that the composition of homomorphisms in \eqref{h1} is in fact injective. When the compact
complex manifold $L$ is a complex projective variety, then
$H^1(L,\, {\mathcal O}_L)/H^1(L,\, {\mathbb Z})$ is in fact an abelian variety.

As $L$ moves over the family $M$, these cocompact lattices fit together to produce
a $C^\infty$ submanifold of the complex manifold ${\mathcal Y}$ in \eqref{e6}.

The Gauss--Manin connection on $R^1q_*\underline{\mathbb C}\,\longrightarrow\, M$ evidently preserves
the above bundle of cocompact lattices $R^1q_*\underline{\mathbb Z}\,\subset\,R^1q_*\underline{\mathbb C}$.
From this it follows immediately that the above $C^\infty$ submanifold $R^1q_*\underline{\mathbb Z}\,\subset\,
{\mathcal Y}$ is in fact a complex submanifold.

Taking fiber-wise quotients, we conclude that
\begin{equation}\label{e7}
{\mathcal A}\, := \,{\mathcal Y}/(R^1q_*\underline{\mathbb Z})\, \longrightarrow\, M
\end{equation}
is a holomorphic family of compact complex tori over $M$.

\begin{remark}\label{rem1}
Let $Y$ be a compact K\"ahler manifold. Consider the short exact sequence of sheaves on $Y$ given by the
exponential map
$$
0\, \longrightarrow\, \underline{\mathbb Z} \, \longrightarrow\, {\mathcal O}_Y
\, \stackrel{\lambda\mapsto \exp(2\pi\sqrt{-1}\lambda)}{\longrightarrow}\, {\mathcal O}^*_Y
\, \longrightarrow\, 0\, ,
$$
where ${\mathcal O}^*_Y$ is a multiplicative sheaf of holomorphic functions with values in
${\mathbb C}\setminus\{0\}$. For the corresponding long exact sequence of cohomologies
$$
H^1(Y,\, \underline{\mathbb Z}) \, \longrightarrow\,
H^1(Y,\, {\mathcal O}_Y) \, \longrightarrow\, H^1(Y,\, {\mathcal O}^*_Y) \, \stackrel{c_1}{\longrightarrow}\,
H^2(Y,\, \underline{\mathbb Z})\, ,
$$
where the connecting homomorphism $c_1$ sends any holomorphic line bundle
$\xi\, \in\, H^1(Y,\, {\mathcal O}^*_Y)$ to $c_1(\xi)$, the quotient
$$
H^1(Y,\, {\mathcal O}_Y)/H^1(Y,\, \underline{\mathbb Z})
$$
gets identified with the Picard group $\text{Pic}^0(Y)$ that parametrizes the topologically trivial
holomorphic line bundles on $Y$.

Consequently, the quotient $\mathcal{A}$ in \eqref{e7}
is naturally identified with the moduli space
${\rm Pic}^0_{{\mathcal Z}/M}$ of topologically trivial holomorphic line bundles on the
fibers of $q\, :\, {\mathcal Z}\, \longrightarrow\, M$. In other words, $\mathcal{A}$
parametrizes all pairs of the form $(L,\, \xi)$, where $L\,\in\, M$, and $\xi$ is a
topologically trivial holomorphic line bundle on $L$.
\end{remark}

Recall that we have the holomorphic symplectic form $d\theta$ on ${\mathcal
Y}$, where $\theta$ is constructed in \eqref{e5}.

\begin{remark}
Recall from Remark \ref{rem0} that $d\theta$ does depend on the K\"ahler form
$\omega$ on $X$.
\end{remark}

\begin{proposition}\label{proposi1}
The $2$-form $d\theta$ on ${\mathcal Y}$ descends to the quotient space
${\mathcal A}$ in \eqref{e7}, or in other words, $d\theta$ is the pullback of a $2$-form
on $\mathcal A$.
\end{proposition}

\begin{proof}
The proof of the proposition is very similar to the proof of Theorem \ref{theo1}.
As before, the local coordinate functions on $M$ constructed in \cite{Hi2} play a crucial
role. We omit the details of the proof.
\end{proof}

Let
\begin{equation}\label{qa2}
q_{\mathcal A}\, :\, {\mathcal Y}\, \longrightarrow\, {\mathcal A}\, :=
\,{\mathcal Y}/(R^1q_*\underline{\mathbb Z})
\end{equation}
be the quotient map (see \eqref{e7}). Let
$\Theta_{\mathcal A}\, \in\, H^0({\mathcal A},\, \Omega^2_{\mathcal A})$ be the
holomorphic symplectic form given by Proposition \ref{proposi1}, so
\begin{equation}\label{ta22}
q^*_{\mathcal A}\Theta_{\mathcal A}\,=\, d\theta\, ,
\end{equation}
where $q_{\mathcal A}$ is the map
in \eqref{qa2}.

The projection $\gamma$ in \eqref{e6} clearly
descends to a map
\begin{equation}\label{vp2}
\varpi_0\, :\, \mathcal{A}\, \longrightarrow\, M\, .
\end{equation}

Note that the isomorphism between $\mathcal{A}$ and ${\rm Pic}^0_{{\mathcal Z}/M}$ in
Remark \ref{rem1} takes $\varpi_0$ to the forgetful map
$$
{\rm Pic}^0_{{\mathcal Z}/M}\, \longrightarrow\, M
$$
that forgets the line bundle, or in other words, it sends any $(L,\, \xi)\,\in\,
{\rm Pic}^0_{{\mathcal Z}/M}$ to the complex Lagrangian $L$ forgetting the line bundle
$\xi$.

\begin{lemma}\label{lem2}
The projection $\varpi_0$ in \eqref{vp2} defines a
completely integrable structure on $\mathcal{A}$ for the symplectic
form $\Theta_{\mathcal A}$.
\end{lemma}

\begin{proof}
Recall that $\mathcal Y$ is holomorphically identified with $\Omega^1_M$ by the map
$\chi$ in \eqref{e4}, and $\chi$ takes the Liouville symplectic form on $\Omega^1_M$ to the
symplectic form $d\theta$ on $\mathcal Y$. Therefore, from \eqref{qa} and \eqref{ta22} we conclude that
$\mathcal A$ is locally isomorphic to $\Omega^1_M$ such that the projection $\varpi_0$ is taken to the
natural projection $\Omega^1_M\, \longrightarrow\, M$, and the symplectic form $\Theta_{\mathcal A}$
on $\mathcal A$ is taken to the Liouville symplectic form on $\Omega^1_M$. The lemma follows
immediately from these.
\end{proof}

\section*{Acknowledgements}

We thank the referee for helpful comments.
The first author is partially supported by a J. C. Bose fellowship.
The second author is supported by
Ministerio de Ciencia e Innovaci\'on of Spain
(grants MTM2016-79400-P, PID2019-108936GB-C21, and
ICMAT Severo Ochoa project SEV-2015-0554) and CSIC
(2019AEP151 and \textit{Ayuda extraordinaria a Centros de Excelencia
Severo Ochoa} 20205CEX001) The third author is supported by CMUP, financed by national funds
through FCT -- Funda\c{c}\~ao para a Ci\^encia e a Tecnologia, I.P., under the project with
reference UIDB/00144/2020. The second and third authors wish to thank
TIFR Mumbai for hospitality and for the excellent conditions provided.