\section{Introduction}

Robot-assisted surgery, with a short but remarkable chronicle~\cite{moustris2011evolution}, has dramatically extended the dexterity and overall capability of surgeons, and plays an important role in modern minimally invasive surgery.
Robotic systems enable precise control, efficient manipulation and vivid observation for the surgical procedures, yielding rich sources of information~\cite{guthart2000intuitive}.
Intelligent understanding of such complex surgical procedure is highly desired for facilitating cognitive assistance. To this end, automatic gesture recognition is fundamentally required for supporting higher-level perception such as surgical decision making~\cite{maier2017surgical}, surgical skill assessment~\cite{poursartip2018analysis} and surgical task automation~\cite{nagy2019dvrk} towards the next generation of operating theatres.
However, accurately recognizing on-going surgical gesture is challenging, due to the complex multi-step actions, frequent state transitions, disturbance in sensor data, various manipulation habits and proficiency of different surgeons.

To address above challenges in automatic surgical gesture recognition, a set of methods have been developed in the past decade. Some methods were based on processing sequential robotic kinematics data (e.g., the position and velocity of the tool tips), using traditional machine learning methods such as variants of hidden Markov models~\cite{tao2012sparse,varadarajan2009data}, linear classifiers with hand-crafted metrics~\cite{zappella2013surgical} and recent deep learning methods such as long short-term memory (LSTM)~\cite{dipietro2016recognizing}, multi-task recurrent neural network~\cite{van2020multi} and multi-scale recurrent network (offline)~\cite{gurcan2019surgical}.
In the meanwhile, purely video based solutions have been intensively explored in the recent years, employing deep convolutional neural networks for extracting high-quality visual features.
Promising gesture recognition results have been achieved relying on temporal convolutional network (TCN)~\cite{lea2017temporal}, recurrent convolutional network~\cite{jin2017sv}, 3D convolutional network~\cite{funke2019using} and symmetric dilated convolution (offline)~\cite{zhang2020symmetric} to extract representative visual features.
However, all these solutions only adopted a single source of information,
without considering the multi-modal joint knowledge inherent in kinematics and visual data
synchronously recorded in robotic systems.

\begin{figure*}[t]
  \centering
  \includegraphics[width=0.93\textwidth]{figures/MRG-Net.pdf}
  \vspace{-5mm}
  \caption{The overview of our proposed multi-modal relational graph network for surgical gesture recognition in robot-assisted surgery.}
  \label{fig1}
  \vspace{-6mm}
\end{figure*}

As we understand, the kinematics and video data can be regarded as the hands and eye of the surgical robot, with eye giving visual guidance information for two hands collaboratively conducting specific actions, while hands drive changes in the visual scene.
In this regard, there are complementary information and joint knowledge contained in the kinematics and video data which are crucial to help gesture recognition.
Several recent works have attempted to develop multi-modal learning methods.
For instance, some unsupervised multi-modal methods have been proposed to handle the problem of time-consuming annotation~\cite{murali2016tsc,zhao2018fast}.
Lea \textit{et al.}~\cite{lea2016learning} designed a latent convolutional skip-chain conditional random field model with variables of scene-based features and kinematics data.
The work of Fusion-KV~\cite{qin2020temporal} learned individual networks for each modality, and combined their predictions through weighted voting at an output level. Qin \textit{et al.}~\cite{qin2020davincinet} further improved Fusion-KV with an attention-based LSTM decoder to predict the surgical state using concatenated multi-modal features.
Despite gaining performance improvement, these multi-modal feature fusions seem straightforward. How to dynamically integrate the multiple sources of information in latent feature space, to reveal and leverage the underlying relationships inherent in kinematics sequences and video scenes, is important yet still remains underexplored.

Recently, graph neural networks have been increasingly receiving research interest, due to their capability to model non-Euclidean relationships among entities~\cite{dwivedi2020benchmarkgnns,scarselli2008graph,zhou2018graph}. Graph convolutional networks (GCN)~\cite{kipf2016semi} have widely demonstrated promising performances on applications in various domains including image classification~\cite{wang2018zero}, neural machine translation~\cite{marcheggiani2018exploiting}, social relationship understanding~\cite{wang2018deep}, etc.
Specifically for robotic surgery related scenarios, there have been pilot studies applying graph neural networks for tool detection in surgical videos~\cite{wang2019graph},
3D point cloud classification~\cite{weibel2019robust,weibel2019addressing} and surgical activity recognition from robotic joint pose estimation~\cite{sarikaya2020towards}.
These achievements inspired us to explore the potential of graph learning for modeling distinct multi-modal data recorded in robotic surgery.

In this paper, we propose a novel \textbf{m}ulti-modal \textbf{r}elational \textbf{g}raph \textbf{net}work (i.e., MRG-Net) to effectively exploit important yet complex relationships in robotics visual and kinematics data for accurate surgical gesture recognition.
Specifically, we first extract the high-level embeddings from video scenes and kinematics sequences with temporal convolutional networks and LSTM units. Then, we leverage relational graph convolutional network to incorporate complementary sources of information and model the underlying multiple types of relations.
Our main contributions are summarized as follows:

\begin{itemize}
	\item We, for the first time, propose a novel online relational graph learning based framework to exploit the joint information with useful relationships in video and kinematics data for accurate surgical gesture recognition.
	\item We evaluated our proposed method on the public robotic surgery dataset JIGSAWS, and set new state-of-the-art \\
	results on both suturing and knot typing tasks, showing the efficacy of combined usage of visual and kinematics information for robotic intelligence.
	\item We have extensively validated our method on in-house datasets collected from da Vinci Research Kit (dVRK) platforms in two centers (i.e., CUHK and JHU) with consistent promising results achieved, demonstrating the general effectiveness of our proposed method.
\end{itemize}

\section{Methods}

In robot-assisted surgery, the robotic system can generate video frames from endoscopy and kinematics sequences from multiple robotic arms, which are later synchronized to the video timestamps. The overview of our proposed network is shown in Fig.~\ref{fig1}.
Our network consists of three components, i.e. visual and kinematic information extraction modules, as well as the relational graph convolutional network. We first extract the visual and kinematic embeddings with visual and kinematic information extraction modules, and then model the complementary information and integrate the informative joint knowledge of these multi-modal features with relational graph convolutional network. As a whole, MRG-Net forms a multi-input single-output design to predict the probability distributions of surgical gestures at each time step.

\vspace{-1mm}
\subsection{Visual and Kinematic Embeddings Extraction}

The first part of the network is the visual and kinematic information extraction modules which extract representative descriptors from each of the following streams respectively: the video frames and the kinematics sequences of robotic left and right arms.
Regarding the visual information, for each time step $t$, current video frames (RGB image) $I_t$ is forwarded to a standard CNN backbone (in this case we leverage a 18-layer deep residual network (ResNet-18)~\cite{he2016deep}), yielding a vector of spatial feature ${u}_t$. For the entire video sample, the series of $\{u_t\}_{t=1}^T$ are input to a temporal convolution module, which adopts an encoder-decoder operation to hierarchically capture relationships across frames at multi-time-scales, yielding stronger spatio-temporal video features of $\{s_t\}_{t=1}^T$.

For the kinematics data, our feature extractor incorporates TCN and LSTM in parallel for modeling the complex sequential information of physical elements and for better capturing the local and longer-term temporal dependencies.
Specifically, the input to TCN stacks the kinematics variables from all time steps, followed by temporal convolutions, pooling, channel-wise normalization and upsampling to encode kinematics features as $\{k_t^{tcn}\}^T_{t=1}$. Meanwhile, LSTM obtains the feature $k^{lstm}_t$ of the current step, by inputting a sequence of kinematics of all its previous steps, for capturing the long-term dependencies in motions. Then, the $k_t^{tcn}$ and $k_t^{lstm}$ are averaged to represent the kinematics feature as $k_t$. Note that we separately encode left and right kinematics as $\{k_t^{l},k_t^{r}\}_{t=1}^T$, since the two robotic arms may conduct different actions and serve for specific purposes in surgery.

\subsection{Fusion of Multi-modal Embeddings with Graph Learning}

Next, a designed graph learning module is subsequently adopted to fuse the above extracted high-level embeddings $\{s_t, k_t^{l}, k_t^{r}\}$ of each time step.
These features have already gained temporal information within each source of time-series data. The following key issue is to effectively exploit their joint knowledge by imposing structured interactions between multi-modal features for accurate gesture recognition.
Intuitively, the graph learning layer plays a role for differentiable message passing framework~\cite{gilmer2017neural}.
Specifically, we denote our graph as $G \! = \! \{\mathcal{V},\mathcal{E}, \mathcal{R}\}$ with nodes $v_i \! \in \! \mathcal{V}$ and edges  $(v_i, r, v_j) \in \mathcal{E}$ where $r\in \mathcal{R}$ is a relation. As shown in Fig.~\ref{fig1}, there are three node entities corresponding to video, left kinematics and right kinematics, whose associated feature descriptors of $\{h_1,h_2,h_3\}$ are initialized as $\{s_t, k_t^{l}, k_t^{r}\}$. These descriptors are then updated by aggregating messages from neighboring nodes with a parameterized propagation rule, which can be generally written as:
\begin{equation}
h_i^{(l+1)} = \sigma\left( \sum_{j\in \mathcal{N}_i} f_m(h_i^{(l)}, h_j^{(l)})\right),\\
\end{equation}
where $h_i^{(l)}$ is the hidden state of node $v_i$ in the $l$-th graph network layer, $\mathcal{N}_i$ is the set of indices of all nodes which are connected with node $i$, the $f_m(\cdot, \cdot)$ denotes the function for accumulating incoming messages from a relational neighbor, and $\sigma(\cdot)$ is the element-wise non-linear activation, i.e., the ReLU in our model.

Intuitively, such interactive feature fusion is important to impose a learnable message passing process for the nodes which have some relations with each other.
Given that the multiple data sources from robotic surgery contain plenty of complementary information, effectively digging the inherent useful relationships among them is difficult while critical to boost the performance of gesture recognition.

\subsection{Multi-relation Modelling in Graph Latent Space}

In our scenario of robot-assisted surgery, there are at least three important types of relations in the video and kinematics data. Specifically, the first is the \emph{vision-to-motion} relation which can be understood as the human's perception with the ``eyes" to provide guidance information for the ``hands" to move. Inversely, the second relation is \emph{motion-to-vision} which reflects the mechanism of ``hands" giving feedback to ``eyes" and also resembles hand-eye coordinates projection of robotic vision~\cite{tsai1989new}. The last relation is \emph{in-between-motions} of left and right arms, which can be considered as two ``hand" assisting each other to complete a task.
The widely-used conventional graph convolutional network~\cite{kipf2016semi} is inadequate to handle various different types of relations, given that its undirected graph with $|\mathcal{R}| \! = \! 1$ is insufficient to model multi-relations of nodes. Instead, we leverage the more powerful relational graph learning scheme,
so that our $G$ is a directed graph endowing a higher capacity for modeling multiple types of directed edges between nodes. In this way, the parameterized propagation function $f_m(\cdot,\cdot)$ in Eq.(1) becomes relation-specific, where the forwarding message update to node $i$ from a relational node $j$ is elaborated as $c_{i,r}  h_j^{(l)}W_r^{(l)}$. The $W_r^{(l)}$ represents a trainable transformation matrix that is uniquely associated to one certain type of relation $r \in \mathcal{R}$.
In other words, different relation types use individual matrices, and only directed edges of the same relation type share their weights.
The parameter $c_{i,r}$ is a normalization constant that correlates to the structure of the graph.
In this way, the layer-wise propagation is achieved by accumulating the message updates through a normalized sum of all neighbor nodes under all relation types.

Hierarchically, we stack two such relational graph learning layers, with each single layer having its separate set of projection weights $\{W_r^{(l)}\}_{l=0}^1$. No deeper layers are added to alleviate the over-smooth problem~\cite{li2018deeper} of GCN, and our preliminary experiments also evidenced that additional layers yielded worse performance yet with heavier computations.
After feature interactions, the final output representation associated with each node is computed by:

\begin{scriptsize}
\begin{equation}
h_i^{(\text{out})} = \sigma\left(\sum_{r\in \mathcal{R}}\sum_{j\in N_i^r} {c_{i,r}}
\sigma\left(\sum_{r\in \mathcal{R}}\sum_{j\in N_i^r}{c_{i,r}}h_j^{(0)} W_r^{(0)}\right) W_r^{(1)}\right),
\label{rgcn2}
\end{equation}
\end{scriptsize}
where the $\mathcal{N}_i^r$ denotes the set of neighbor indices of node $i$ under a relation type $r \in \mathcal{R}$. Specifically in our model, we identify three different types of relations with $|\mathcal{R}|=3$.
For instance, as shown in Fig.~\ref{fig1}, the video node $h_1$ receives messages from kinematics nodes $\{h_2,h_3\}$, both under the relation type of \emph{motion-to-vision} (cf. $W_2^{(0)}$ in brown arrow).
The left kinematics node $h_2$ receives messages from video node $h_1$ under relation type of \emph{vision-to-motion} (cf. $W_1^{(0)}$ in purple arrow), and from right kinematics $h_3$ under relation type of \emph{in-between-motions} (cf. $W_3^{(0)}$ in red arrow).
The weight $c_{i, r}$ is heuristically set as $1/|\mathcal{N}_i^r|$.
We exclude the self-loop aggregation for each node, with the consideration that, for our graph classification task, the self-loop information tends to result in feature redundancy during the update process, weakening the messages propagated from neighbor nodes (i.e., other modalities). Note that such a practice does not cause knowledge leakage since the hierarchical propagation can compensate those self-contained information through iterative interactions among nodes.
In addition, the regularization strategy of basis decomposition~\cite{schlichtkrull2018modeling} is applied for $\{W_r^{(l)}\}_{l=0}^1$ to prevent rapid growth of the number of parameters with multi-relational data.

\subsection{Overall Loss Function}

After interacting the multi-modal information in latent space for capturing joint knowledge, the relational graph learning layers produce updated representations for the nodes. Recall that the hidden state for each node represents the descriptors for each modality at time step $t$, so we rephrase $\{h_i^{(\text{out})}\}_{i=1}^3$ into $\{\tilde{s}_t, \tilde{k}_t^l, \tilde{k}_t^r\}$ as the set of features for video, left and right kinematics after graph learning. They are concatenated to convey the joint knowledge, and forwarded to a fully-connected layer for obtaining the classification prediction $\hat{p}_t$ for each frame:

\begin{equation}
 \hat{p}_t=Softmax(\textbf{concat}[\tilde{s}_t, \tilde{k}_t^l, \tilde{k}_t^r] W_{\text{fc}} + b).
\end{equation}
With the situation in the robotic surgery that the duration of each conducted gesture varies widely (e.g., the gesture of ``loosening more suture" occupies only about 1\% time on average of the whole suturing task, while ``pushing needle through tissue" takes up almost 30\% task duration), we use the weighted cross-entropy loss to combat such inter-class imbalance for training samples.
Denoting $\alpha$ as the class balancing weight, $\Theta$ as MRG-Net parameters of all trainable layers,
we optimize the overall loss function:
\begin{equation}
\mathcal{L}(\mathcal{X,Y}; \Theta) = \frac{1}{T}\sum\nolimits_{t} -\alpha \cdot \log \hat{p}_t,
\end{equation}
where $\mathcal{X}$ is the multi-modal input space, $\mathcal{Y}$ denotes the gesture categories.

\subsection{Implementation Details}
Overall, the entire framework composing of the relational graph layers and the separate video and kinematics feature extractors is trained end-to-end.
The encoder and decoder of TCN backbone consists of $3$ temporal convolutional layers with $\{64,96,128\}$ filters for encoder and $\{96,64,64\}$ filters for decoder, with the kernel size of $51$.
For the visual information, we first train the CNN backbone (ResNet-18) using video sequences, then we generate the spatial-CNN features $u_t \! \in \! \mathbb{R}^{128}$ from the pretrained backbone to train the whole model more efficiently. For kinematic data, we first convert the rotation matrix
into Euler angles,
then normalize all the data to zero mean and unit variance. The relational graph layers have 64-dimensional hidden states and output states, with dropout (rate $\!=\!0.2$) applied.

Our graph learning framework is implemented with the Deep Graph Library (DGL)~\cite{wang2019dgl} in PyTorch with an NVIDIA Titan Xp GPU. The video frames are resized to resolution 320*256 with random crop 224*224 to reduce the training parameters and prevent over-fitting.
We used the Adam~\cite{kingma2014adam} optimiser with learning rate of $5e^{-3}$ and weight decay of $5e^{-4}$ to train the proposed network. The training process took around 3 hours for a hundred epochs.
To avoid the case of coincidence, we trained the same model for three times and reported their average results.

\section{Experiments}

\subsection{Public Dataset and Evaluation Metrics}

We first extensively validate our proposed MRG-Net on the public dataset of JIGSAWS~\cite{gao2014jhu} (JHU-ISI Gesture and Skill Assessment Working Set) on two tasks (i.e., suturing and knot typing), which consist of 39 videos and 36 videos respectively alongside with kinematics data of left and right robotic arms from the \emph{da Vinci} surgical system.
The kinematics sequences include position, rotation matrix, linear velocity, rotational velocity of tool tip and angles of gripper.
A total of ten categories of surgical gestures are annotated for each single frame in suturing task and six in knot typing task (cf.~\cite{gao2014jhu} for detailed gesture definitions).
Our experimental setting adopts \emph{leave-one-user-out} cross validation, following the practice of previous works on this benchmark~\cite{ahmidi2017dataset}.
The employed evaluation metrics on JIGSAWS dataset include: i) Accuracy (\%) in the frame-wise level, which is to calculate the percentage of correctly recognized frames, ii) Edit Score~\cite{lea2016segmental}
(in range $[0,100]$, the higher score the better), which is designed to measure the performance in video segmentation level for emphasizing temporal smoothness.

\subsection{Comparison with Other State-of-the-art Methods}
\begin{table}[t]
\begin{center}
\caption{Results of different methods on JIGSAWS Suturing dataset for gesture recognition.
}\label{tab1}
\scalebox{0.94}{
\begin{tabular}{|c|c|c|c|c|}
\hline
\multirow{2}{*}{Methods} & \multicolumn{2}{|c|}{Input data} & \multirow{2}{*}{Accuracy}& \multirow{2}{*}{Edit Score} \\
\cline{2-3}
~ & ~Kin & Vid &&\\
\hline
TCN~\cite{lea2016temporal} & \checkmark  && 79.6 & 85.8\\
Forward LSTM~\cite{dipietro2016recognizing} & \checkmark  && 80.5 $\pm$ 6.2 & 75.3\\
TricorNet~\cite{ding2017tricornet} & \checkmark  & & 82.9 & 86.8\\
Bidir. LSTM~\cite{dipietro2016recognizing} & \checkmark  && 83.3 $\pm$ 5.7 & 81.1\\
Bidir GRU~\cite{dipietro2019segmenting} & \checkmark  && 84.7 $\pm$ 6.0 & 88.5\\
APc~\cite{van2020multi} & \checkmark  & & 85.5 & 85.3\\
\hline
TCN~\cite{lea2016temporal} &  & \checkmark & 81.4 & 83.1\\
Policy+Value~\cite{gao2020automatic} &  & \checkmark & 81.7 & 88.5\\
3D CNN(K)+window~\cite{funke2019using} & & \checkmark & 84.3 & 80.0\\
\hline
LC-SC-CRF~\cite{lea2016learning} & \checkmark & \checkmark & 83.5 & 76.8\\
Fusion-KV~\cite{qin2020temporal} & \checkmark & \checkmark & 86.3 & 87.2\\
\bfseries MRG-Net (Ours) & \checkmark & \checkmark &\bfseries 87.9 $\pm$ 4.2 &\bfseries 89.3 $\pm$ 5.2\\
\hline
\end{tabular}
}
\end{center}
\vspace{-4mm}
\end{table}

\begin{figure}[t]
    \centering
    \includegraphics[width=0.4\textwidth]{figures/colorbar.pdf}
    \vspace{-3mm}
    \caption{Color-coded ribbon illustration of surgical gesture recognition on Suturing task (a) and Knot Typing task (b) with ground truth (top) and our results (bottom).} \label{fig3}
    \vspace{-5mm}
\end{figure}

\begin{table}[t]
\begin{center}
\caption{Results of different methods on JIGSAWS Knot Typing dataset for gesture recognition.}\label{tab_knot}
\scalebox{1}{
\begin{tabular}{|c|c|c|c|c|}
\hline
\multirow{2}{*}{Methods} & \multicolumn{2}{|c|}{Input data} & \multirow{2}{*}{~Accuracy~}& \multirow{2}{*}{Edit Score} \\
\cline{2-3}
~ & Kin & Vid &&\\
\hline
SC-CRF~\cite{ahmidi2017dataset} & \checkmark &  & 78.9 & N/A \\
BoF~\cite{ahmidi2017dataset} & & \checkmark & 86.5 & N/A \\
MsM-CRF~\cite{ahmidi2017dataset} & \checkmark & \checkmark & 77.3 & N/A \\
\bfseries MRG-Net (Ours) & \checkmark & \checkmark &\bfseries 88.1 $\pm$ 3.8 &\bfseries 87.0 $\pm$ 6.8\\
\hline
\end{tabular}
}
\end{center}
\vspace{-4mm}
\end{table}

\begin{figure}[t]
\centering
\includegraphics[width=0.25\textwidth]{figures/test.png}
\vspace{-3mm}
\caption{Bar chart of gesture-wise recognition accuracy.}
\label{fig2}
\vspace{-6mm}
\end{figure}

We compare our proposed MRG-Net with previous state-of-the-art methods on the benchmark dataset and we report the mean accuracy and mean Edit Score with standard deviation (std.) (those results without std. mean the original paper didn't report them).
 These methods are grouped into \emph{purely kinematics based} (i.e., SC-CRF~\cite{ahmidi2017dataset}, TCN~\cite{lea2016temporal}, Forward and Bidirectional LSTM~\cite{dipietro2016recognizing}, Bidirectional GRU~\cite{dipietro2019segmenting}, TricorNet~\cite{ding2017tricornet} of hybrid TCN and LSTM, APc~\cite{van2020multi} using multi-task RNN), \emph{purely video based} (i.e., TCN~\cite{lea2016temporal}, Policy+Value~\cite{gao2020automatic} of offline reinforcement learning, 3D CNN with post-processing~\cite{funke2019using}), and \emph{multi-modal based} methods (i.e., LC-SC-CRF~\cite{lea2016learning} and MsM-CRF~\cite{ahmidi2017dataset} with traditional machine learning, BoF~\cite{ahmidi2017dataset} with manual extracted features and Fusion-KV~\cite{qin2020temporal} with deep learning).

For the suturing task, which is the most popular task with more samples and gestures in JIGSAWS, we compare our results with eleven state-of-the-art methods listed in Table~\ref{tab1}.
We first see that our MRG-Net significantly outperforms the state-of-the-art uni-modal methods, with the accuracy exceeding the previous best kinematics based method~\cite{van2020multi} by 2.4\% and best video method~\cite{funke2019using} by 3.6\%. With multi-modal learning to capture the complementary information of visual and motion data, improved results are obtained, with the LC-SC-CRF~\cite{lea2016learning} outperforming six of the uni-modal methods on accuracy and the Fusion-KV~\cite{qin2020temporal} outperforming all of them. Importantly, our MRG-Net achieves the highest accuracy of 87.9\% (with lowest std. 4.2\%) and Edit Score of 89.3 (with lowest std. 5.2) compared among the multi-modal methods, demonstrating the superiority of our method enabling dynamic interactions for modeling the inherent relations of multiple input sources with graph learning.

For the knot typing task, which is more complex than suturing while less validated in previous works, we list the results in Table~\ref{tab_knot}. The performance of current state-of-the-art uni-modal and multi-modal methods are referenced from the benchmark in~\cite{ahmidi2017dataset}.
It can be observed that traditional multi-modal method (MsM-CRF),
if without sufficient integration of the visual and kinematics information in the complex task,
even obtained worse performance compared with pure video based method and pure kinematics based method.
Leveraging our proposed relational graph learning method to interact the visual and kinematics embeddings in the latent space, a high recognition accuracy of 88.1\% can be achieved on this task.

For qualitative results, Fig.~\ref{fig3} illustrates the visualization results on both suturing and knot typing tasks in the form of color-coded ribbon, demonstrating the temporal consistency and smoothness of the surgical gesture predictions leveraging the high-quality multi-modal representations.

\subsection{Ablation Analysis on Our Method}

\begin{table}[t]
\begin{center}
\caption{Ablation study on key components of our method using the same backbone on JIGSAWS Suturing dataset.}
\label{tab2}
\scalebox{1}{
\begin{tabular}{|c|c|c|c|c|}
\hline
\multirow{2}{*}{Methods} & \multicolumn{2}{|c|}{Input data} & \multirow{2}{*}{~Accuracy~}& \multirow{2}{*}{Edit Score} \\
\cline{2-3}
~ & Kin & Vid &&\\
\hline
Pure-Vis & & \checkmark & 81.7 $\pm$ 6.7 & 86.5 $\pm$ 6.9\\
Pure-Kin & \checkmark & & 82.6 $\pm$ 6.5 & 86.6 $\pm$ 7.5\\
TCN-KV (w/o split) & \checkmark & \checkmark & 86.1 $\pm$ 5.6 & 85.3 $\pm$ 7.1\\
TCN-KV & \checkmark & \checkmark & 86.2 $\pm$ 5.4 & 86.1 $\pm$ 6.4\\
GCN-KV & \checkmark & \checkmark & 86.8 $\pm$ 4.9 & 87.4 $\pm$ 6.5\\
\bfseries MRG-Net & \checkmark & \checkmark & \bfseries 87.9 $\pm$ 4.2 &  \bfseries 89.3 $\pm$ 5.2\\
\hline
\end{tabular}
}
\end{center}
\end{table}

\begin{figure}[t]
\centering
\vspace{-6mm}
\includegraphics[width=0.32\textwidth]{figures/exp.pdf}
\vspace{-2mm}
\caption{Embeddings of pre- (left) and post- (right) relational graph message propagation multi-modal features (blue: vision, red: left kinematics, pink: right kinematics). Best viewed in color.}
\label{fig4}
\vspace{-7mm}
\end{figure}

To validate the contribution of each key component in our proposed MRG-Net,
Table~\ref{tab2} lists the results of five ablation studies implemented with our own backbone on suturing task for direct comparison:
1) Pure-Vis: uni-modal using visual data,
2) Pure-Kin: uni-modal using kinematics,
3) TCN-KV (w/o split): merging video and kinematics (without splitting left and right arms) with TCN,
4) TCN-KV: merging video and kinematics (splitting left/right arms) with TCN,
5) GCN-KV: multi-modal learning with plain GCN without multi-relation,
and finally our proposed multi-relational MRG-Net.

We see that fusing visual and kinematics features in latent space can provide richer knowledge for achieving higher performance, even using simple concatenation of representations in temporal convolutional networks.
Note that splitting left and right kinematics data yields better results than treating both kinematics as a whole (comparing 3rd/4th rows with TCN), which also reveals the different information contained in the left and right ``hands" of robotic systems.
Moreover, our graph learning for interactive multi-modal message passing can bring improvement over TCN.
Further modelling multi-relations as designed based on domain knowledge, the gesture recognition performance gets higher, which confirms the significance of considering the ``edges" between ``nodes" diversely, as they incorporate distinct types of relations among various information sources.

In addition, we analyze the detailed accuracy across gesture category, as shown in Fig.~\ref{fig2}.
We notice a large variance in the results with the highest accuracy achieving 93\% (G1 ``reaching for needle with right hand"), while the lowest being less than 10\% (G10 ``loosening more suture").
The performance imbalance may be still due to the large variance in gesture frequency and sample numbers, which reflects the challenges in this recognition task which remains to be further conquered in future research.
Besides, we see that our relational graph multi-modal learning consistently outperforms Pure-Vis and Pure-Kin by a large margin (especially for G9 ``using right hand to help tighten suture" with strong visual/motion relationships), demonstrating the stable effectiveness of our method.
Last but not least, Fig.~\ref{fig4} visualizes the node features from MRG-Net learning process with t-SNE~\cite{maaten2008visualizing}, where the left and right embed the sets of $\{s_t,k_t^l,k_t^r\}$ and $\{\tilde{s}_t, \tilde{k}_t^l, \tilde{k}_t^r\}$, respectively, for observing feature clusters before and after multi-relational graph updates.
It clearly shows that multi-modal features are harmoniously fused from interactive message propagation and aggregation.

\begin{figure}[t]
  \centering
  \includegraphics[width=0.45\textwidth]{figures/peg_transfer_demo.pdf}
  \vspace{-3mm}
  \caption{(a) Gestures of peg transfer, (b) Color-coded ribbon illustration of surgical gesture recognition on peg transfer task with ground truth (top) and our results (bottom).}
  \label{fig_peg}
  \vspace{-5mm}
\end{figure}

\subsection{Experiment on In-house dVRK Dataset from Two Centers}

To further validate our method, we have collected robotic multi-modal datasets on dVRK platforms in two centers from CUHK and JHU. The data collection conditions of the two datasets had variations in settings of
hand-eye calibrations, illuminations, operating locations, which reflected complications in real-world practice.
Both kinematics sequences and camera videos have been recorded and synchronized.

To build the in-house dVRK datasets, we experimented on the peg transfer task (see Fig.~\ref{fig_peg} (a)), which is one of the most popular tasks present in Fundamentals of Laparoscopic Surgery~\cite{ritter2007design} and widely adopted for surgical skill training~\cite{joseph2010chopstick}. Specifically, we defined and manually annotated five different gestures for peg transfer:
A1: Idle (No action performed);
A2: Reach for peg (with left hand);
A3: Lift peg (with left hand);
A4: Exchange (transfer the peg to right hand);
A5: Place peg (with right hand).
The dataset consists of 24 sequences with 12 sequences from CUHK and JHU each. The duration of sequences is within the range of 20-60 seconds due to different length settings to transfer the peg.
Within each site, all the operation records were performed by the same user
who is familiar with using dVRK platform. The kinematics data includes the position/orientation of end-effector and opening angle of the gripper, at the meanwhile, videos are synchronously recorded and there are all down-sampled to 10Hz in pre-processing.

Considering different conditions in data acquisition such as hand-eye settings of dVRK systems, appearances of peg transfer boards, we individually trained and tested models for each dataset, in which we split each dataset to perform 3-fold cross-validation (8 sequences for training and 4 for testing).
We adopted the same evaluation metrics as JIGSAWS (i.e., accuracy and Edit Score).
The results are listed in the Table~\ref{tab_peg_cuhk} and Table~\ref{tab_peg_jhu}, in which Baseline means the standard 2D CNN backbone used in MRG-Net (ResNet-18), while Pure-Vis and Pure-Kin represent the same configurations as Table~\ref{tab2}, which adopted TCN and LSTM.

It can be observed that, on both datasets, compared to 2D based Baseline, the methods of Pure-Vis and Pure-Kin can obtain higher accuracies and Edit Scores, leveraging their consideration of temporal information in the sequential data.
On both the datasets, our proposed MRG-Net achieves the highest accuracy and Edit Score, consistently outperforming the Baseline, Pure-Vis and Pure-Kin methods with a notable margin. The Edit Scores reaches as high as 98.7\% on CUHK dataset and 96.4\% on JHU dataset, which reflects the good smoothness and stability of the prediction on dVRK data. In addition, we notice that the model performances for CUHK dataset are overall slightly higher than that for JHU dataset.
We analyze that this is related to the variation of task duration between these two sites. The data recorded from JHU present larger diversity regarding task completion speed compared to CUHK data, thus may be more challenging for recognition. It will be interesting and valuable to further investigate model behavior differences between two datasets in our future work.

\begin{table}[t]
\begin{center}
\caption{Results of different methods on Peg Transfer dataset in site CUHK for gesture recognition.}\label{tab_peg_cuhk}
\scalebox{1}{
\begin{tabular}{|c|c|c|c|c|}
\hline
\multirow{2}{*}{Methods} & \multicolumn{2}{|c|}{Input data} & \multirow{2}{*}{~Accuracy~}& \multirow{2}{*}{Edit Score} \\
\cline{2-3}
~ & Kin & Vid &&\\
\hline
Baseline(ResNet-18) &  & \checkmark & 80.7 $\pm$ 7.4 & 35.1 $\pm$ 8.6\\
Pure-Vis &  & \checkmark & 88.9 $\pm$ 2.8 & 96.7 $\pm$ 3.9\\
Pure-Kin & \checkmark &  & 89.2 $\pm$ 2.5 & 95.6 $\pm$ 3.7\\
\bfseries MRG-Net (Ours) & \checkmark & \checkmark &\bfseries 91.0 $\pm$ 2.1 &\bfseries 98.7 $\pm$ 3.4\\
\hline
\end{tabular}
}
\end{center}
\vspace{-2mm}
\end{table}

\begin{table}[t]
\begin{center}
\caption{Results of different methods on Peg Transfer dataset in site JHU for gesture recognition.}\label{tab_peg_jhu}
\scalebox{1}{
\begin{tabular}{|c|c|c|c|c|}
\hline
\multirow{2}{*}{Methods} & \multicolumn{2}{|c|}{Input data} & \multirow{2}{*}{~Accuracy~}& \multirow{2}{*}{Edit Score} \\
\cline{2-3}
~ & Kin & Vid &&\\
\hline
Baseline(ResNet-18) &  & \checkmark & 78.5 $\pm$ 8.2 & 21.3 $\pm$ 9.8\\
Pure-Vis &  & \checkmark & 83.0 $\pm$ 3.6 & 95.1 $\pm$ 4.2\\
Pure-Kin & \checkmark &  & 85.1 $\pm$ 3.4 & 95.5 $\pm$ 3.9\\
\bfseries MRG-Net (Ours) & \checkmark & \checkmark &\bfseries 87.3 $\pm$ 2.9 &\bfseries 96.4 $\pm$ 3.6\\
\hline
\end{tabular}
}
\end{center}
\vspace{-7mm}
\end{table}

\section{Conclusion and Future Work}
\label{CONCLUSIONS}
This paper presents a novel online multi-modal graph learning method to dynamically integrate complementary information in video and kinematics data from robotic systems, to achieve accurate surgical gesture recognition. Multi-relational representation aggregation is achieved through a designed directed graph to capture the underlying
joint knowledge
between the visual scenes and kinematics motions.
The effectiveness of our method is validated with state-of-the-art performance on the public dataset of JIGSAWS on two tasks of suturing and knot typing. Meanwhile, we investigate the significance of each component in our network by conducting ablation studies on JIGSAWS suturing dataset.
Furthermore, the proposed method is validated on our collected in-house dVRK datasets, shedding light on the general efficacy of our approach.

In our future work, we shall explore
how to resolve the data variance and domain gap due to different acquisition environments and hardware platforms of our two in-house datasets. Potentially, we will design a 6-DOF transformer (a trainable homogeneous mapping) to uniformly align kinematics data from different platforms to a common feature space, and rely on optical flow to tackle the visual gap among different environment.
With the help of these methods, we can improve the generalization ability of our method, so as to make full use of more robotic surgery datasets and achieve a cross-platform training and testing scheme. Moreover, we will investigate how to extract the multi-modal embeddings with unsupervised learning schemes in order to reduce the annotation cost.
We will also apply the developed visual-kinematics based surgical gesture recognition to downstream scenarios such as sub-task automation for robotic surgery.

\newpage
\bibliographystyle{IEEEtran}


\section*{Comment on `Open is not forever: a study of vanished open access journals'}
Gerta R\"ucker\\
Institute of Medical Biometry and Statistics, Medical Center -- University of Freiburg\\
Stefan-Meier-Strasse 26\\
D-79115 Freiburg, Germany\\
e-mail: ruecker@imbi.uni-freiburg.de

\vspace{1cm}

This is a comment to an article by Laakso, Matthias and Jahn (arXiv:2008.11933).

\vspace{1cm}

With interest I read an article published by Laakso et al. \cite{laakso2020opena} that was already commented on by Shelomi \cite{shelomi2020commenta}. Laakso et al. reported a study in which they found 176 Open Access (OA) journals that had vanished from the web between 2000 and 2019. As Shelomi noted, it was unfortunate that Laakso et al. did not distinguish between predatory and reputable journals, and I agree. In my present comment, however, I want to point to another weakness of the article by Laakso et al., which is the statistical analysis. They consider their sample of 176 vanished journals and present some descriptive statistics on their characteristics, such as the time lag between the last publication and vanishing (their Figure 3 and Table 3), the distribution of academic disciplines (their Figure 4), their affiliation (whether academic or not), and the geographic distribution (Figure 5 and Table 4). Of course, this is legitimate as a description of a selected sample. However, the authors, ignoring that their sample was selected conditionally on a journal having vanished from the web, misinterpret their findings. Their statement that `our study provides valuable insight into the types of OA journals that are especially at risk of vanishing' (page 22) is not justified by evidence, simply because a risk can only be quantified on a population basis. Otherwise, any analysis can only be descriptive.

Laakso et al. found that `North America and South Asia represent a disproportionately larger share of vanished than active OA journals' and interpreted this as `journals published in North America \dots belong to the high risk group'. For the geographic distribution this is justified, as they refer to a list of active OA journals as a control group (Table 4). However, they use the term `risk' also for other factors, writing for example: `Moreover, our study provides valuable insight into the types of OA journals that are especially at risk of vanishing', then referring to journals affiliated with academic institutions or scholarly societies, without providing the numbers of active OA journals for comparison. This might provoke a well known and common error: the inversion fallacy, that is, confusing the denominators. To find out whether scholarly-based journals are at higher risk of vanishing than other journals, it is not correct to compare the proportions of journals by affiliation \emph{within vanished journals}. It would be necessary to compare vanished and active journals and to adjust for covariates in a multivariable analysis. The limitation of this study as a case series should have been acknowledged by the authors.


\section{\label{sec:intro}Introduction}

Over the years, structural changes have been intensively investigated in the region between the semi-magic $^{68}$Ni and the doubly-magic $^{78}$Ni~\cite{Sorlin2002, 78NiPRL, Bissell2016, 78Ninature, Wraith2017, DeGroote2017}. The investigation involves multiple experimental methods as well as theoretical models, which provide various nuclear properties (masses, spins, life-times, transition probabilities, excitation energies, moments and radii), aiming to get a global view of the nuclear structure in this region. As studies deepen, more interesting nuclear phenomena appear. Some examples are: collective effects occur around and above $N = 40$~\cite{Xie2019, Cheal2010, Aoi2010, Chiara2011}; the proton $p_{3/2}$ and $f_{5/2}$ orbitals invert as neutrons are filling the neutron $g_{9/2}$ orbital due to the tensor part of the monopole interaction~\cite{Flanagan2009, Cheal2010, Wraith2017}; a spherical shape coexists with a deformed state around the neutron closed shells \mbox{$N = 40, 50$~\cite{CRIDER2016108, Yang2018, Yang2016}}; the indication of a weak sub-shell effect observed at $N = 40$ disappears quickly with more protons added above $Z = 28$~\cite{Bissell2016, Xie2019}.

\begin{figure*}[!t]
\includegraphics[width=0.9\textwidth]{Fig1.pdf}
\vspace{-4mm}
\caption{\label{fig:COLLAPS} A schematic view of the COLLAPS experimental setup and the laser frequency-mixing and frequency-doubling system. For details see text.}
\vspace{-5mm}
\end{figure*}

The nuclear properties of ground and isomeric states, such as spins, moments and charge radii, have made significant contributions in elucidating the above-mentioned nuclear phenomena. This was achieved by laser spectroscopy measurements on the isotopic chains of nickel ($Z =28$), copper ($Z =29$), zinc ($Z =30$) and gallium ($Z =31$)~\cite{Flanagan2009, Cheal2010, Vingerhoets2010, Procter2012, Bissell2016, DeGroote2017, Wraith2017, Xie2019, Simon2020, Yang2016, Yang2018, DeGroote2020}. Nuclear spins and magnetic dipole moments are sensitive probes of the single-particle nature or configuration mixing of the wave functions~\cite{Flanagan2009, Cheal2010, Wraith2017, DeGroote2017}, while the electric quadrupole moment and charge radii tell us more about the nuclear shape and collectivity~\cite{Yang2018, Bissell2016, Yang2016}.

Deformation, triaxiality and shape coexistence have been observed for zinc isotopes, which have also been reported for germanium isotopes around $N = 40$ based on $\gamma$-spectroscopy and reaction experiments~\cite{Xie2019, Ayangeakaa2016, Sun2014, Toh2013}. Germanium ($Z = 32$) has four protons outside the $Z = 28$ major shell closure, which may induce additional correlations leading to a complex wavefunction and collective effects for the low-lying states of germanium isotopes around $N = 40$~\cite{Ayangeakaa2016, Heyde2011, Heyde2016}. This collective feature can be further investigated with the nuclear moment measurements of germanium isotopes around $N = 40$ in combination with wave function calculations using large-scale shell-model interactions.

Until now, collinear laser spectroscopy has not been applied to study germanium isotopes, due to the fact that (1) germanium species cannot be easily produced at ISOL facilities, and (2) the suitable fine structure transitions are not easily accessible with standard laser equipment. In this article, we report the first hyperfine structure (hfs) measurement in the $4s^2 4p^2 \, ^3P_1 \rightarrow 4s^2 4p 5s \, ^3P_1^o$ transition of $^{69,71,73}$Ge atoms by combining the frequency-mixing and frequency-doubling laser techniques. State-of-the-art relativistic Fock-Space Coupled-Cluster (FSCC) calculations of the atomic hyperfine fields and electric field gradients have been performed, showing a good agreement with the experimental values, and thus benchmarking these atomic theories. The calculations also reveal that incorrect nuclear moments have been reported for $^{69}$Ge from earlier atomic beam magnetic resonance studies~\cite{Oluwole1970}. The extracted nuclear moments are compared with large-scale shell-model calculations using the JUN45 interaction in the $f_{5/2}pg_{9/2}$ model space in order to understand the collective effects. The systematic comparison of the nuclear moments of the 9/2$^{+}$ states in zinc ($Z = 30$), germanium ($Z = 32$) and selenium \mbox{($Z=34$)} allow the investigation of the structural evolution from single-particle to collective effects around $N = 40$ as more protons are added above the $Z = 28$ shell closure.

\section{Experimental procedure}

The experiment was performed at the collinear laser spectroscopy setup, COLLAPS~\cite{Neugart2017}, located at ISOLDE-CERN.
The radioactive germanium isotopes were produced by a 1.4-GeV proton beam impinging on a $\textrm{ZrO}_2$ target with a sulphur leak. The sulphur vapour was introduced into the target material where the sulphur atoms bind to form volatile GeS molecules.  These molecules easily diffuse out of the target and then disassociate under electron impact to form germanium ions inside a plasma source. The germanium ions were accelerated to 50\;keV, mass separated using the high-resolution isotope separator (HRS) and subsequently cooled and bunched in a gas-filled linear Paul trap (ISCOOL)~\cite{Mane2009}. The accumulated ions were released in short bunches with a typical temporal width of 5 $\mu$s every 5 ms. Due to the large isobaric contamination in all of the Ge beams \cite{Koster2003}, the accumulation time was optimized at 5~ms to avoid the overfilling of the ISCOOL.

\begin{table*}[!t]
\caption{\label{tab:AandB}%
Magnetic and electric hfs constants ($A$ and $B$) for both atomic states ($4s^2 4p^2 \, ^3P_1$  and $4s^2 4p 5s \, ^3P_1^o$) obtained from this work. The numbers from Refs.~\cite{Oluwole1970, Childs1963, Childs1966} are also summarized as a comparison.}
\renewcommand{\arraystretch}{1.2}
\begin{ruledtabular}
\begin{tabular}{cccccccccc}
$A$ & $N$ & $I^\pi$ & $T_{1/2}$ & $A_{\rm l}^{\rm lit}$ (MHz) & $A_{\rm l}$ (MHz) & $A_{\rm u}$ (MHz) & $B_{\rm l}^{\rm lit}$ (MHz) & $B_{\rm l}$ (MHz) & $B_{\rm u}$ (MHz) \\
\colrule
69 & 37 & 5/2$^-$ & 39.05(10) h & $\mp$23.40(3)~\cite{Oluwole1970} & $-$29.0(7) & $+$494.6(9) & $\pm$8.28(8)~\cite{Oluwole1970} & $+$32(2)\footnotemark[1] & $-$21(3)\footnotemark[1] \\
71 & 39 & 1/2$^-$ & 11.43(3) d & $-$87.005(3)~\cite{Childs1963} &  & $+$1460.8(16) &  &  &  \\
73 & 41 & 9/2$^+$ & stable & $+$15.5480(18)~\cite{Childs1966} &  & $-$262.2(12) &  $-$54.566(9)~\cite{Childs1966} & & $+$40(6)
\footnotetext[1]{Correlation C($B_{\rm l}$, $B_{\rm u}$) = -0.445}
\end{tabular}
\end{ruledtabular}
\end{table*}

As shown in \mbox{Fig.~\ref{fig:COLLAPS}}, the bunched ion beam was then deflected into the COLLAPS beamline, where it was neutralized in-flight by passing through a charge exchange (CE) cell filled with sodium vapor. The state $4s^2 4p^2 \, ^3P_1$ of the germanium atom was populated in the neutralization process \cite{Vernon2019}. A frequency fixed continuous-wave (cw) laser beam, overlapped with the germanium beam in a collinear geometry, was used to probe the $4s^2 4p^2 \, ^3P_1 - 4s^2 4p 5s \, ^3P_1^o$ \mbox{(269.13411 nm)} transition of the germanium atom. By applying a varying voltage to the germanium ions before entering the CEC, neutralized germanium atoms were resonantly excited to the $4s^2 4p 5s \, ^3P_1^o$ state through Doppler tuning. The emitted fluorescence photons from the resonantly excited atoms were detected as a function of the tuning voltage, by the use of 4 photomultiplier tubes (PMTs) \cite{Kreim2014}. A software time gate was applied in order to select photons only when each bunch of germanium atoms passes and de-excites in front of the PMTs. This resulted in a reduction of the background from laser stray light, non-resonantly scattered photons and PMT dark counts by a factor of about $10^3$.

\begin{figure}[!t]
\includegraphics[width=0.5\textwidth]{Fig2.pdf}
\vspace{-4mm}
\caption{\label{fig:HFSandTOFheatmap}Color-coded 2D plot with time vs frequency (voltage step) vs counts acquired for $^{73}$Ge with the new DAQ system (TILDA) of COLLAPS. The two components presented in the TOF spectrum are corresponding to the germanium atom bunch (gate 1 in red) and the possible molecular contamination bunch (gate 2 in green). Gating on the germanium atom bunch allows the hfs spectrum to be reconstructed with a significantly reduced background.}
\vspace{-4mm}
\end{figure}

To take full advantage of ISCOOL and to retain as much information on the beam structure in the recorded data as possible, a new data acquisition system (DAQ) was introduced at COLLAPS. It was developed at the TRIGA-SPEC setup in Mainz \cite{Kaufmann.2015} and here we report its online-operation. The main advantage of this DAQ is the time-resolved photon detection. In order to reduce the photon background, the detection of photons has to be restricted to the time window when the bunched beam is traversing the detection region. Previously, this was realized by hardware gating, which had to be adapted in advance for each isotope according to the time-of-flight (TOF) recorded. With the new DAQ, photons are recorded with a time stamp relative to the ion extraction trigger of ISCOOL. This is shown for $^{73}$Ge in Fig.~\ref{fig:HFSandTOFheatmap}(a): the $x$-axis represents the scanning voltage, converted into frequency, while the $y$-axis is the TOF since the ISCOOL extraction pulse. The number of photons detected within a 500-ns interval during $n$-time extractions from ISCOOL are color-coded. These numbers are integrated along a specified time in Fig.~\ref{fig:HFSandTOFheatmap}(b), which provides the time structure of the  bunch. Integration along a fixed frequency reveals the hfs resonance spectrum of the isotope, as illustrated in Fig.~\ref{fig:HFSandTOFheatmap}(c). The spectra in \mbox{Fig.~\ref{fig:HFSandTOFheatmap}(c)-(d)} are obtained by selecting events within the specific time window indicated with dashed horizontal lines in Fig.~\ref{fig:HFSandTOFheatmap}(a) and with their corresponding colored dashed lines in Fig.~\ref{fig:HFSandTOFheatmap}(b).

\begin{figure}[!t]
\includegraphics[width=0.5\textwidth]{Fig3.pdf}
\vspace{-4mm}
\caption{\label{fig:hfs}Hyperfine structure spectra of $^{69,71,73}$Ge in the $4s^2 4p^2 \, ^3P_1 \rightarrow 4s^2 4p 5s \, ^3P_1^o$ transition. The red lines show the best fit with a Voigt line profile. Note that the production of the known short-lived isomers of $^{71,73}$Ge (T$_{1/2}$($^{71m}$Ge) = 20.41(18)~ms and T$_{1/2}$($^{73m}$Ge) = 499(11)~ms) \cite{Abusaleem2011, Singh2019} is at least 100 times lower than that of the ground state \cite{Bissell:2157172}. Thus, these isomers could not be observed in this experiment.}
\vspace{-4mm}
\end{figure}

In Fig.~\ref{fig:HFSandTOFheatmap}(a) three regions can be clearly distinguished. Outside of the dashed lines, the background is dominated by stray laser light, which is weak compared to the more intense signal within the time gate of the bunch. One can clearly see that the bunch is separated into two parts, the first one arriving after about 52\,$\mu$s, while the second one is delayed by another 5\,$\mu$s. The time interval between two bunches released from ISCOOL is about 1000 times longer. Thus, the two consecutive \lq bunches' seen in Fig.~\ref{fig:HFSandTOFheatmap}(a) are from photons emitted from two different species. Those with a heavier mass arrive later in front of the PMT's. In the first time window (gate 1) clear resonance spots are seen, as well as an additional beam-related but frequency-independent background (see also Fig.~\ref{fig:HFSandTOFheatmap}(c)). This background is attributed to the de-excitation of all contaminant neutral particles that were excited in the CE process. The second part of the bunch (gate 2) does not exhibit any resonance-like structures, only beam-related photon background, as seen in Fig.~\ref{fig:HFSandTOFheatmap}(d). It is assumed that this beam-related photon background is due to laser-light scattering from the molecular contamination in the beam and de-excitations of these molecules excited via the CE process.

With the latest DAQ, these individual parts of the bunch can be clearly separated in online and offline analysis. Therefore, by gating on the first component (gate 1) instead of the plotted range (40\,$\mu$s) of the TOF spectrum, the hfs spectrum of $^{73}$Ge was obtained with a much improved signal-to-background ratio, as shown in Fig.~\ref{fig:HFSandTOFheatmap}(c) and Fig.~\ref{fig:HFSandTOFheatmap}(e), respectively.

One challenge for the laser spectroscopy measurement of germanium atoms is the production of the 269 nm cw laser light. It requires frequency-doubling of the wavelength 538 nm. This wavelength is lying in the \lq green gap' region, which is not covered by the commonly used continuous wave (cw) Ti:Sa and dye laser systems. To bridge this wavelength gap, a frequency mixing method was employed. As shown in \mbox{Fig.~\ref{fig:COLLAPS}}, a 824 nm laser beam from a Ti:Sa laser cavity (Sirah Matisse-2) and a 1550 nm laser beam generated by a single-frequency fiber laser (Koheras Boostik E 15) are superimposed and single-pass through a periodically poled crystal in a frequency mixing unit (Sirah MixTrain) to generate the sum frequency of 538 nm ($\frac{1}{\lambda}=\frac{1}{\lambda_1}+\frac{1}{\lambda_2}$). This light is then coupled into a frequency doubling unit (Sirah WaveTrain) to produce 269 nm light with an output power around 90 mW. A small fraction of the output light from the frequency mixing unit was locked to a wavelength meter (HighFinesse WSU10), which was regularly calibrated with a stabilized diode laser (Toptica DLPRO780) locked to the $F = 2  \rightarrow F = 3$ hyperfine transition of the D1 line in $^{87}$Rb.

\section{Results}

\begin{figure}[!t]
\includegraphics[width=0.5\textwidth]{Fig4.pdf}
\vspace{-5mm}
\caption{\label{fig:AandB} Ratio of the $A$ (a) and $B$ hyperfine constants (b) for the $4p^2 \, ^3P_1$ (lower) and $4p5s \, ^3P_1^o$ (upper) atomic states, deduced with two fitting procedures. One with the $A_{\rm l}$ and $B_{\rm l}$ values for each isotope fixed (green) to the high-precision value reported in literature \cite{Oluwole1970, Childs1963, Childs1970}, and one with all hyperfine constants as free fit parameters (black). The hyperfine constants ratios calculated with atomic relativistic FSCC are shown by red dashed lines.}
\vspace{-5mm}
\end{figure}

The obtained hfs spectra of $^{69,71,73}$Ge isotopes, as shown in Fig.~\ref{fig:hfs}, were fitted with a $\chi^2$-minimization Python routine using the SATLAS package~\cite{Gins2018}. A Voigt profile with one side peak was used to compensate for the slightly asymmetric resonance peaks, which resulted from the energy loss due to the population of higher atomic states or the collision-excitation in the charge exchange process~\cite{Neugart1977, Bendali1986}. All the recorded hfs spectra of each individual isotope were fitted simultaneously, generating one optimal reduced $\chi^2$, with magnetic dipole and electric quadrupole hyperfine constants ($A$ and $B$) as common fit parameters for each isotope.

\begin{table}[!b]
\caption{\label{tab:HFSratios}%
Deduced $R_{\rm A} = A_{\rm u} / A_{\rm l}$ and $R_{\rm B} = B_{\rm u} / B_{\rm l}$ for $^{69,71,73}$Ge isotopes
with the $A_{\rm l}$ and $B_{\rm l}$ fixed to literature values or with all hyperfine constants as free fit parameters.}
\renewcommand{\arraystretch}{1.2}
\begin{ruledtabular}
\begin{tabular}{c|cc|cc}
 & \multicolumn{2}{c|}{$R_{\rm A}$} & \multicolumn{2}{c}{$R_{\rm B}$} \\
 & All free & $A_{\rm l}$ fixed to lit. & All free & $B_{\rm l}$ fixed to lit. \\
\colrule
$^{69}$Ge & $-$17.1(4) & $-$21.21(4) & $-$0.64(11) & $-$2.8(4) \\
$^{71}$Ge & $-$17.3(3) & $-$16.79(2) & & \\
$^{73}$Ge & $-$16.3(5) & $-$16.87(8) & $-$0.70(17) & $-$0.73(11)
\end{tabular}
\end{ruledtabular}
\end{table}

The $A$ and $B$ parameters for the lower atomic state ($4s^2 4p^2 \, ^3P_1$),  named $A_{\rm l}$ and $B_{\rm l}$ in the following, are reported in literature with high precision for each of the three \mbox{odd-$A$} germanium isotopes from atomic beam magnetic resonance experiments~\cite{Childs1963, Childs1966, Oluwole1970, Childs1970}, as summarized in \mbox{Tab.~\ref{tab:AandB}}. We therefore fitted our data in two ways: (1) with the $A_{\rm l}$ and $B_{\rm l}$ values fixed to the literature value and upper hyperfine constants as free fit parameters for each isotope, and (2) with all lower and upper hyperfine constants as free fit parameters. The ratio of the hyperfine parameters of the upper and lower atomic states ($R_{\rm A}$ = $A_{\rm u}$/$A_{\rm l}$, $R_{\rm B}$ = $B_{\rm u}$/$B_{\rm l}$) reflect the ratio of the atomic hyperfine fields of both levels. These ratios should be the same for all isotopes, apart from a possible small hyperfine anomaly that can cause some scatter of the order of at most a few percent in $R_{\rm A}$. In \mbox{Table~\ref{tab:HFSratios}}, as well as in \mbox{Fig.~\ref{fig:AandB}}, the ratio of $R_{\rm A}$ and $R_{\rm B}$ resulting from the two fit procedures are shown. When using the first fitting procedure, both the $A$ and $B$ factor ratios for $^{69}$Ge deviate strongly from those observed for the other isotopes, suggesting a problem with the reported hyperfine parameters of $^{69}$Ge \cite{Oluwole1970}, from which the literature nuclear moments have been deduced. When all four hyperfine parameters are free fit parameters, the deduced ratios of $R_{\rm A}$ and $R_{\rm B}$ are consistent for all three isotopes, as shown in \mbox{Tab.~\ref{tab:HFSratios}} and \mbox{Fig.~\ref{fig:AandB}}. To corroborate these conclusions, state-of-the-art atomic calculations were performed, as discussed in the following section.

\begin{table*}[!t]
\caption{\label{tab:AtomicTheory}%
Atomic $A_0$ and $q$ parameters extracted from $^{73}$Ge experimental results and calculated with atomic relativistic FSCC method.}
\renewcommand{\arraystretch}{1.2}
\begin{ruledtabular}
\begin{tabular}{ccccccc}
At. state & $A^{\rm exp}$ (MHz) & $A^{\rm exp}_{\rm 0}$ (MHz) & $A^{\rm th}_{\rm 0}$ (MHz) & $B^{\rm exp}$ (MHz) & $q^{\rm exp}$ (MHz/b) & $q^{\rm th}$ (MHz/b) \\
\colrule
$4p^2 \, ^3P_1$ & $+$15.5480(18) \cite{Childs1966} & $-$79.667(10) & $-$74(6) & $-$54.566(9) \cite{Childs1966} & $+$278.4(14) & $+$277(8) \\
$4p^2 \, ^3P_2$ & $-$64.4270(7) \cite{Childs1966} & $+$330.12(2) & $+$321(11) & $+$111.825(13) \cite{Childs1966} & $-$571(3) & $-$564(17) \\
$4p5s \, ^3P_1^o$ & $-$262.2(12)\footnotemark[1] & $+$1343(6) & $+$1314(45) & $+$40(6)\footnotemark[1] & $-$204(31) & $-$208(13) \\
\end{tabular}
\footnotetext[1]{Hyperfine parameters measured in this work.}
\end{ruledtabular}
\end{table*}

\begin{table*}[!t]
\caption{\label{tab:MandQ}%
Magnetic dipole and electric quadrupole moments of $^{69, 71, 73}$Ge isotopes compared with JUN45. The numbers from Refs.\cite{Oluwole1970, Childs1966, Makulski2006, Kello1999} are also summarized as a comparison.}
\renewcommand{\arraystretch}{1.2}
\begin{ruledtabular}
\begin{tabular}{ccccccccc}
$A$ & $N$ & $I^\pi$ & $\mu^{\rm lit}$ ($\mu_N$) & $\mu^{\rm exp}$ ($\mu_N$) & $\mu^{\rm JUN45}$ ($\mu_N$) & $Q_{\rm s}^{\rm lit}$ (b) & $Q_{\rm s}^{\rm exp}$ (b) & $Q_{\rm s}^{\rm JUN45}$ (b) \\
\colrule
69 & 37 & 5/2$^-$ & 0.735(7)\cite{Oluwole1970} & $+$0.920(5) & $+$1.048 & $+$0.027(5)\cite{Oluwole1970} & $+$0.114(7) & $+$0.150 \\
71 & 39 & 1/2$^-$ & $+$0.54606(7) \cite{Childs1966} & $+$0.547(5) & $+$0.438 &  &  &  \\
73 & 41 & 9/2$^+$ & $-$0.87824(5) \cite{Makulski2006} & -0.904(21)\footnotemark[1]  & $-$0.955 & $-$0.196(1) \cite{Kello1999} & -0.198(4)\footnotemark[1] & $-$0.258 \\
\end{tabular}
\footnotetext[1]{Weighted mean of the moments extracted from the measured hyperfine parameters and calculated hyperfine fields of the 3 transitions in \mbox{Tab.~\ref{tab:AtomicTheory}}.}
\end{ruledtabular}
\end{table*}

\subsection{Atomic hyperfine field calculations }
To further investigate the aforementioned discrepancy between the hyperfine constants from literature and our experimental results for $^{69}$Ge, we performed atomic calculations to obtain the hyperfine fields of three atomic fine structure levels ($4s^24p^2$ $^3P_1$, $4s^24p^2$ $^3P_2$ and $4s^24p5s$ $^3P_1^o$), for which experimental information is available for the stable $^{73}$Ge isotope~\cite{Childs1966}.

High-quality treatment of relativistic and electron correlation effects is required to obtain accurate and reliable computational results. Therefore, the multireference relativistic FSCC method \cite{Kaldor1998, Visscher2001} is employed for the investigation of the electric field gradients (EFGs) ($q^{\rm th}$) and the hyperfine magnetic field constants ($A_0^{\rm th}$). This method was shown to be one of the most powerful approaches for treatment of spectra and properties of heavy many-electron atoms \cite{ABcoupledcluster}.

The FSCC method requires a closed shell reference state from which the ground state and excited states can be reached by adding or removing electrons. Neutral germanium has an open shell ground state electron configuration [Ar]$3d^{10}4s^24p^2$ and thus the closed shell Ge$^{2+}$ system ([Ar]$3d^{10}4s^2$) is used as the reference state. The ground state and excited states of interest are reached by adding two electrons to the corresponding virtual orbitals, which comprise the model space. The intermediate Hamiltonian (IH) approach is applied to avoid the intruder-state problem \cite{Landau2001}. The finite field method \cite{Pople1968} is used to determine the $q^{\rm th}$ and $A_0^{\rm th}$ properties, as described in Refs. \cite{Yakobi2007, Haase2020}. All calculations are carried out in the framework of the Dirac-Coulomb Hamiltonian and the nuclear charge distribution is modeled by a Gaussian function as described in Ref. \cite{Visscher1997}. The $q^{\rm th}$ calculations were carried out using the DIRAC15 program package \cite{DIRAC15}, while DIRAC17 was used for the $A_0^{\rm th}$ calculations \cite{DIRAC17}. The final values are obtained using the full 4-component DC Hamiltonian and the relativistic core-valence 4-zeta basis set of Dyall \cite{Dyall2006} (cv4z), augmented by three diffuse functions in each symmetry in an even-tempered fashion. A large model space was used, consisting of the $4p$ $5s$ ($4d$ $5p$ $6s$ $4f$ $5d$ $6p$ $7s$ $5f$ $5g$ $7p$ $6d$ $7d$ $8p$ $6g$ $8s$ $6f$) orbitals, where the orbitals in parentheses are in the intermediate space $P_i$. All the electrons were correlated and virtual orbitals with energies up to 500 a.u. were included in the calculation. The finite field perturbation strength $\lambda$ was $\num{1e-6}$ for the $q^{\rm th}$ calculations and $\num{1e-4}$ for $A_0^{\rm th}$.

To estimate the uncertainty on the calculated values, we have investigated the effect of various computational parameters: the error from the limited size of the basis set, model space and virtual space, while also estimating the contribution from higher order excitations and the relativistic Gaunt term. These sources of error are combined by assuming them to be independent to give a total conservative uncertainty estimate for each calculated property. The uncertainty of $q^{\rm th}$ is 3.0 \% for the $4p^2$ ${}^3P_1$ and $4p^2$ states and 6.4 \% for $4p5s$ ${}^3P_1^o$, while for $A_0^{\rm th}$ the uncertainties are 7.8 \%, 3.5 \% and 3.4 \% respectively for these three states. The final recommended values and uncertainties for $q^{\rm th}$ and $A_0^{\rm th}$ are shown in \mbox{Tab.~\ref{tab:AtomicTheory}}. Further details on the procedure used for the calculation of these properties and the uncertainties can be found in Ref. \cite{Gustafsson2020}.

The calculated atomic observables, $A_0$ and $q$, are proportional to the magnetic hyperfine field ($B_0$) created by the electrons at the point of the nucleus, \mbox{$A_0 = B_0 / J$}, and to the electric field gradient ($V_{zz}$) created by the electronic cloud, \mbox{$q = e V_{zz}$}, respectively. Here, $J$ is the atomic spin. These atomic observables are related to the measured $A$ and $B$ parameters as follows: \mbox{$A_0 = A I / \mu$} and \mbox{$q = B / \, Q_{\rm s}$}, with $\mu$ and $Q_{\rm s}$ the nuclear magnetic dipole moment and electric quadrupole moment, respectively, and $I$ the nuclear spin. From the experimental hyperfine constants of the three atomic states taken from Refs. \cite{Childs1966} and measured in this work, as well as the experimental magnetic and quadrupole moments from \mbox{Refs.~\cite{Kello1999, Makulski2006}}, we can calculate the experimental $A_0$ and $q$ parameters for $^{73}$Ge. These can be directly compared with atomic theory calculations, as presented in \mbox{Tab.~\ref{tab:AtomicTheory}}. A remarkable agreement is achieved for both $A_0$ and $q$, for all three states within less than 5\%. We further compared the ratios ($R_{\rm A}$, $R_{\rm B}$) of hyperfine constants of two atomic states ($4p^2 \, ^3P_1$ and $4p5s \, ^3P_1^o$) measured in this work with the atomic calculation, as shown in \mbox{Fig.~\ref{fig:AandB}}. The ratios of the hyperfine constants from theory show a good agreement with experimental values for all three isotopes of $^{69,71,73}$Ge obtained with fitting procedure (2). This further supports our experimental result for the $A_{\rm l}$ and $B_{\rm l}$ hyperfine constants of $^{69}$Ge.

\subsection{Hfs constants and nuclear moments}
As mentioned above, both our experimental results and the atomic calculations are inconsistent with the literature hyperfine constants for the $4p^2 \, ^3P_1$ atomic state of $^{69}$Ge. A revision of the lower state hfs parameters of $^{69}$Ge is therefore suggested. From our measurements, the upper-state hyperfine parameters, $A_{\rm u}$ and $B_{\rm u}$, can also be determined for the first time for all isotopes, yielding an independent measurement of the nuclear moments. For $^{71,73}$Ge, the final $A_{\rm u}$ and $B_{\rm u}$ are obtained with $A_{\rm l}$ and $B_{\rm l}$ fixed to the literature values~\cite{Childs1963, Childs1966}, since they are known with high-precision. In \mbox{Tab.~\ref{tab:AandB}}, the revised hyperfine parameters of the lower atomic state for $^{69}$Ge, and the newly measured hyperfine constants of the upper atomic state for $^{69,71,73}$Ge are shown together with literature values.

A precise nuclear magnetic moment of $^{73}$Ge has recently been redetermined using gas-phase NMR measurements on GeH$_4$~\cite{Makulski2006} and a precise electric quadrupole moment was extracted from molecular microwave data of GeO and GeS~\cite{Kello1999}. Thus, the magnetic dipole $\mu$ and electric quadrupole $Q_{\rm s}$ moments of $^{69,71}$Ge can be calculated from the experimental $A$ and $B$ parameters in a model independent way relative to those of $^{73}$Ge, by using:
\begin{equation}
\mu= \frac{IA}{I_{\rm ref}A_{\rm ref }} \mu_{\rm{ref}}
\label{EQ1}
\end{equation}
\vspace{-3mm}
\begin{equation}
Q_{\textrm{s}}= \frac{B}{B_{\rm ref}} Q_{\rm s,ref}
\label{EQ2}
\end{equation}

The large $A_{\rm u}$ hyperfine parameter is used to extract the magnetic moments of $^{69,71}$Ge via \mbox{Eq.~(\ref{EQ1})}. Since the hyperfine $B$ parameters for both atomic states are comparable in magnitude, two sets of electric quadrupole moments are extracted from $B_{\rm l}$ and $B_{\rm u}$ using \mbox{Eq.~(\ref{EQ2})}. We use the weighted average of the two quadrupole moments as the final value, after taking into account the correlation between the two hyperfine $B$ parameters \textcolor{red}{\cite{Leo:1987kd}}. The results are summarized in \mbox{Tab.~\ref{tab:MandQ}} together with the literature values.

For the reference isotope $^{73}$Ge, we also determine an independent set of nuclear moments, using the measured hyperfine parameters and calculated hyperfine fields for each of the three atomic states shown in \mbox{Tab.~\ref{tab:AtomicTheory}}. These moments are fully consistent with the literature values (\mbox{Tab.~\ref{tab:MandQ}}), in particular for the quadrupole moment reaching almost similar precision. This illustrates the significant progress that has been made in atomic calculations in the last few years.

\begin{figure}[!b]
\includegraphics[width=0.45\textwidth]{Fig5.pdf}
\vspace{-3mm}
\caption{\label{fig:shells}Proton and neutron orbitals around the $Z, N$ = 28 and the $Z, N$ = 50 major shell closures according the nuclear shell-model. The model space of the effective interaction JUN45 is also marked.}
\end{figure}

\section{Discussion}

\begin{figure*}[!t]
\includegraphics[width=0.98\textwidth]{Fig6.pdf}
\vspace{-3mm}
\caption{\label{fig:GeZnOddEnergy} Ground and low-lying excited (isomeric) states of $^{67-73}$Zn (top) and $^{69-75}$Ge (bottom) compared to results from JUN45 calculations. States with half-lives of at least 10 ns and a firm spin assignment are plotted. Thick lines represent states with half-lives longer than 1 ms.}
\vspace{-4mm}
\end{figure*}

The relevant proton and neutron shells in the nickel mass region are displayed in \mbox{Fig.~\ref{fig:shells}}. For the even-$Z$ elements (like nickel, zinc and germanium), with neutron numbers between $N = 28$ and $N = 50$, the neutrons are expected to dominate the ground state structure of the odd-$A$ isotopes in a naive single particle shell-model picture. By filling the $\nu 2p_{3/2}, \nu 1f_{5/2}, \nu 2p_{1/2}, \nu 1g_{9/2}$ orbitals, this would lead to ground-state (g.s.) spins of 3/2$^-$, 5/2$^-$, 1/2$^-$ and 9/2$^+$ for the odd-$A$ isotopes, respectively. Thus, for $^{69}$Ge and its isotones, $^{67}$Zn and $^{65}$Ni, with \mbox{$N = 37$}, the unpaired neutron is expected to occupy the $\nu f_{5/2}$ orbital, resulting in a g.s. spin of 5/2$^{-}$, which is consistent with the experimental assignment for each of these isotones~\cite{Browne2010, Junde2005, Nesaraja2014}. For isotones with \mbox{$N$ = 39}, a spin 1/2$^{-}$ is then expected and also experimentally confirmed for $^{67}$Ni, $^{69}$Zn and $^{71}$Ge~\cite{Junde2005, Nesaraja2014, Abusaleem2011}. So, below $N = 40$, the single-particle (SP) shell-model filling seems to be respected in germanium, despite having four valence protons, which could induce significant correlations and deformation. Moving to $N = 41$ and beyond, the filling of the $\nu g_{9/2}$ orbital is then beginning and the odd-$N$ isotopes are all expected to have  g.s. spin of 9/2$^{+}$. This is indeed the case for $^{73}$Ge and $^{69}$Ni but not for $^{71}$Zn~\cite{Nesaraja2014, Abusaleem2011, Singh2019}. This might suggest some stabilizing effect from a completely filled $\pi p_{3/2}$ orbital in $^{73}$Ge, and onset of correlations between protons in the open $\pi p_{3/2}$ orbital in $^{71}$Zn.

Recent studies have suggested that the structure of the zinc isotopes between $N = 40$ and $N = 50$~\cite{Wraith2017, Yang2018} presents some complexity with long-lived SP-like or deformed isomers occurring in each of the odd-$N$ isotopes.  For the germanium isotopes, deformation was proposed for the even-$A$ isotopes around $N=40$~\cite{Heyde2011, Heyde2016} but a systematic discussion of the nuclear moments of the odd-$N$ isotopes has not yet been done. We first investigate the nuclear structure around $N=40$ by looking at the low-lying energy levels of $^{67-73}$Zn and $^{69-75}$Ge isotopes, as summarized in \mbox{Fig.~\ref{fig:GeZnOddEnergy}}. Note that only the energy levels with a firm spin assignment and with measured half-lives longer than \mbox{10~ns} are plotted, as nuclear moments have been measured for most of them \cite{Mertzimekis2016}. These energy levels are also compared to large-scale shell-model calculations using the JUN45 effective interaction~\cite{Honma2009} in the $f_{5/2}pg_{9/2}$ model space starting from a $^{56}$Ni core, as illustrated in \mbox{Fig.~\ref{fig:shells}}.

While the shell-model reproduces well the level ordering in germanium and zinc isotopes below \mbox{$N = 40$}, it fails to reproduce the complex level structure in both isotopic chains beyond $N = 40$~\cite{Ayangeakaa2016, Sun2014, Toh2013, Yang2018}. This is apparent from the spins of the low lying states in $^{73, 75}$Ge and $^{73}$Zn, which may not be easily understood from the normal filling of the shell-model orbitals. In the $^{73}$Zn isotope, the isomeric character of the 5/2$^{+}$ state allowed a measurement of its nuclear moments, and an unambiguous assignment of its spins and parity to be made, which was debated before in literature~\cite{Wraith2017}. From the magnetic moment, it was clear that the unpaired neutrons mostly occupy the $g_{9/2}$ orbital, but the large quadrupole moment could be only explained by including proton and neutron excitations across $Z = 28$ and \mbox{$N = 50$}, leading to a triaxial shape~\cite{Yang2018}. Investigating the experimental electromagnetic properties of the low-lying states in the odd-$A$ germanium isotopes will thus help to get a better understanding of their structure, which will be discussed in the following sections.

\subsection{Magnetic Moments}
The magnetic dipole moment, and more specifically the related $g$-factor ($g = \mu/I$), is an excellent probe of the orbital that is occupied by the unpaired particles (in this case neutrons). The available experimental $g$-factors of the ground and isomeric states of $^{69-75}$Ge are compared with the effective SP $g$-factors of relevant orbitals, as presented in \mbox{Fig.~\ref{fig:momentMu}(a)}. The effective SP $g$-factors for the orbitals of $\nu f_{5/2}$, $\nu p_{1/2}$, $\nu g_{9/2}$ are calculated using effective $g$-factors of $g_{s}^{\rm eff}$ = $0.7g_{s}^{\rm free}$ and $g_{l}^{\rm eff}=g_{l}^{\rm free}$, which are the typical values used in the region~\cite{Yang2016, Farooq-Smith2017}.

\begin{figure}[!t]
\includegraphics[width=0.5\textwidth]{Fig7.pdf}
\vspace{-10mm}
\caption{\label{fig:momentMu}(a) $g$-factor of ground and low-lying isomeric states of $^{67-75}$Ge, compared with the shell-model calculations using JUN45 interaction. (b) Experimental $g$-factors of ground and low-lying isomeric 9/2$^{+}$ states of the isotones of even-$Z$, zinc, germanium and selenium isotopes.}
\vspace{-5mm}
\end{figure}

The newly measured magnetic moment (and $g$-factor) of the 5/2$^-$ g.s. of $^{69}$Ge ($N = 37$) is in excellent agreement with the $\nu f_{5/2}$ effective SP value, intuitively pointing to its simple shell-model character of an unpaired valence neutron in the $\nu f_{5/2}$ orbital. The available experimental $g$-factors for the states with spin 9/2$^+$ in $^{69,71,73}$Ge lie close to the effective SP $g$-factor of the $g_{9/2}$ orbital, as shown in \mbox{Fig.~\ref{fig:momentMu}(a)}. Again this suggests a configuration dominated by an unpaired neutron in the $\nu g_{9/2}$ orbital. More interestingly, the \mbox{$g$-factor} of the 9/2$^+$ state of $^{73}$Ge deviates more from the effective SP $g$-factor (enhanced in Fig. \ref{fig:momentMu}(b)), pointing to a possible onset of collective effects from $N=41$ onwards. As for the g.s. of $^{71,75}$Ge, both having a spin/parity of 1/2$^-$, the structure is likely dominated by an odd neutron in the $p_{1/2}$ orbital as the $g$-factors are close to the $\nu p_{1/2}$ SP value, yet deviate somewhat. Similar deviations have been observed for other 1/2$^-$ states, e.g. in the neighbouring zinc isotopes~\cite{Wraith2017}, as well as for heavier isotopes with spin 1/2$^-$ with a valence particle filling the $p_{1/2}$ orbital (e.g. Ag and In)~\cite{Eberz1987}.

These experimental $g$-factors are also compared with the shell-model calculations using the JUN45 interaction, as presented in \mbox{Fig.~\ref{fig:momentMu}(a)}. The calculations nicely reproduce the new experimental value of the 5/2$^-$ g.s. of $^{69}$Ge ($N = 37$) as well as the $g$-factors of the 9/2$^{+}$ states in $^{69-73}$Ge. However, the calculated wave functions for these states are found to be very fragmented. The calculated dominant configuration of the wave function for the 5/2$^-$ g.s. of $^{69}$Ge is about 26\%. For the 9/2$^{+}$ states of $^{69-73}$Ge, the main component of the wave function is, in all cases, less than 50\%. For all the isotopes, the leading configuration is the same as that concluded from the effective SP $g$-factors. Surprisingly, the calculated $g$-factors of the 1/2$^-$ g.s. of $^{71,75}$Ge are close to the SP values, so they also deviate from the experimental values. Further theoretical investigation is needed to understand this deviation for the 1/2$^-$ states.

Earlier studies have shown that the $N = 40$ subshell effect observed in the nickel and copper isotopes, quickly disappeared with more protons added above $Z = 28$, as described by various experimental observables: magnetic and quadrupole moments~\cite{Flanagan2009, DeGroote2017, Wraith2017, Cheal2010}, charge radii~\cite{Bissell2016, Xie2019, Procter2012}, nuclear masses~\cite{Rahaman2007, Guenaut2007}, $E(2^{+})$ excitation energies, and $B(E2)$ transition rates~\cite{Perru2006, Aoi2010, Chiara2011, Louchart2013}. This indicates that an increase in collectivity around and above $N$ = 40 is expected when going away from $Z$ = 28, as has been observed in the zinc and gallium isotopes~\cite{Cheal2010, Yang2018, Wraith2017, Xie2019, Perru2006, Aoi2010, Chiara2011, Louchart2013, Rahaman2007, Guenaut2007}.

In order to have a systematic investigation of the structural evolution from single-particle nature to the collective effect, we compare the experimental magnetic moments of the 9/2$^{+}$ states for even-$Z$ nuclei (zinc, germanium and selenium) around $N = 40$, as shown in \mbox{Fig.~\ref{fig:momentMu}(b)}. The $g$-factors of all the 9/2$^{+}$ states in the region around $N$ = 40 are close to the effective SP $g$-factor, clarifying the leading configuration of an unpaired neutron in the $g_{9/2}$ orbital in their wave functions. However, a continuously increased deviation from the SP value is obvious from zinc, to germanium and selenium, giving a clear sign of the increased collectivity as more protons are added above $Z = 28$ closed shell. This is further confirmed by comparing the contribution of the main configuration of wave functions (single neutron in $g_{9/2}$ orbital), which is about $\sim$35\% for $^{69}$Zn and $\sim$14\% for $^{71}$Ge.

\begin{figure}[!t]
\includegraphics[width=0.5\textwidth]{Fig8.pdf}
\vspace{-5mm}
\caption{\label{fig:momentQs}(a) Quadrupole moments of ground and low-lying isomeric states of $^{67-75}$Ge, compared with the shell-model calculations using JUN45 interaction using effective charges $e_{p}^{\rm eff}$ = $1.5e_{p}$ and $e_{n}^{\rm eff}=1.1e_{n}$. (b) Experimental quadrupole moments of ground and low-lying isomeric 9/2$^{+}$ states of even-$Z$ zinc, germanium and selenium.}
\vspace{-5mm}
\end{figure}

\subsection{Quadrupole moments}
The known experimental quadrupole moments of ground and isomeric states in $^{69-75}$Ge are presented in \mbox{Fig.~\ref{fig:momentQs}(a)}. The new value of the quadrupole moment of the 5/2$^-$ g.s. in $^{69}$Ge is well reproduced by the shell-model calculation. The same agreement can be found for the 9/2$^{+}$ states of $^{71-73}$Ge. It is known from the above-discussed magnetic moments that the main configuration for these 9/2$^+$ states comes from the unpaired valence neutron in the $\nu g_{9/2}$ orbital.

In \mbox{Fig.~\ref{fig:momentQs}(a)}, we see a linear trend in the quadrupole moments of these 9/2$^+$ states, when more neutrons fill the $\nu g_{9/2}$ orbital. A similar trend has been observed in the neighboring zinc isotopes~\cite{Wraith2017, Yang2018}, which are compared to those of germanium in \mbox{Fig.~\ref{fig:momentQs}(b)}. In the case of the zinc isotopic chain, the quadrupole moments of the 9/2$^+$ states in $^{69}$Zn ($N=39$) and $^{79}$Zn ($N=49$) reflect a relatively pure configuration of single neutron particle and single neutron hole in the $\nu g_{9/2}$ orbital, respectively~\cite{Wraith2017, Yang2018}, from which we estimated the SP quadrupole moment of $\nu g_{9/2}$ orbital (see Ref.~\cite{Wraith2017} for more details). A linear trend for quadrupole moments with increased neutron number for a seniority-1 $(\nu g_{9/2})^n$ configuration is then calculated, as shown with the black dash line in \mbox{Fig.~\ref{fig:momentQs}(b)}.

For the Ge isotopes, with only three measured quadrupole moments, it is difficult to make a firm conclusion. It appears that the slope of the curve is somewhat steeper than for the Zn isotopes, which would point to some enhanced deformation (correlations) for these isotopes with four protons outside the $Z=28$ shell. However, more data on neutron-rich isotopes are needed to make a firm conclusion.

\section{Summary}
The hfs of germanium atoms was measured for the first time in the $4s^2 4p^2 \, ^3P_1 \rightarrow 4s^2 4p 5s \, ^3P_1^o$ atomic transition, taking advantage of the development of the laser frequency-mixing technique. A clear discrepancy with literature magnetic and electric hyperfine parameters is observed for $^{69}$Ge. The systematic analysis of the hfs of $^{69,71,73}$Ge isotopes obtained in this work requires a revision of the hyperfine constants for $^{69}$Ge, resulting in new magnetic and quadrupole moments. State-of-the-art atomic relativistic Fock-Space Coupled-Cluster calculations were performed for the hyperfine fields in three atomic fine structure levels, allowing to determine the quadrupole moment of $^{73}$Ge to a precision of 2\%, in agreement with the precision value obtained from molecular theory calculations and experiments.

The available experimental nuclear moments of ground and isomeric states of $^{69-75}$Ge around $N = 40$ are interpreted through a comparison to large-scale shell-model calculations in the $f_{5/2}pg_{9/2}$ model space. A comparison of the $g$-factors with the effective SP \mbox{$g$-factors} reveals the nature of the orbital occupied by the unpaired neutrons. Yet, the calculated wave functions reveal a very mixed configuration. Through a systematic comparison of the nuclear moments of germanium isotopes with its isotones (zinc and selenium) around $N = 40$, an increase in collectivity is observed, when protons are added to $Z=28$, reflecting a sign of structural change from single particle to deformation when moving away from $Z = 28$.

To shed more light on the structural evolution in the region, further study of germanium isotopes up to $N = 50$ and even beyond is necessary, with the well-established laser systems and atomic transition studied in this work. This will allow us to have a better understanding of the structure changes in the nickel region, particularly, the gradual emergence of collectivity. \\

\begin{acknowledgments}
We acknowledge the support of the ISOLDE collaboration and technical teams. We would like to acknowledge the Center for Information Technology of the University of Groningen for their support and for providing access to the Peregrine high-performance computing cluster. This work was supported by the National Key R\&D Program of China (No. 2018YFA0404403), the National Natural Science Foundation of China (No:11875073, U1967201); the BriX Research Program No. P7/12, FWO-Vlaanderen (Belgium), GOA 15/010 from KU Leuven; the UK Science and Technology Facilities Council grants ST/L005794/1 and ST/P004598/1; the NSF grant PHY-1068217, the BMBF Contracts No 05P18RDCIA, the Max-Planck Society, the Helmholtz International Center for FAIR (HIC for FAIR); the EU Horizon2020 research and innovation programme through ENSAR2 (No. 654002);
\end{acknowledgments}