\section{Introduction}
Human pose estimation aims to locate the body joints from image or video data. It has received increasing attention from the computer vision community in the past couple of years and has been utilized in various applications such as animation\cite{animation}, sports\cite{sports,xiang2008intelligent}, and healthcare\cite{healthcare}. Due to the development of deep convolutional neural networks, recent works achieve significant progress on this topic\cite{hourglass, simplebaseline, cpn, hrnet}.

In order to get higher accuracy, many existing methods usually comprise wide and deep network architectures because they could bring powerful representation capacity. This leads to tremendous parameters and a huge number of floating-point operations (FLOPs). However, the most common operating environments for pose estimation applications are resource-constrained edge devices such as smartphones or robots. The incremental computing cost and parameter memory are crucial barriers to the deployment of pose estimation.

There is also some work focus on the lightweight human pose estimation method. Lite-HRNet\cite{litehrnet} follows High-Resolution Net (HRNet)\cite{hrnet} and reduces its parameters and computation cost. Lite-HRNet achieves extremely low parameters and FLOPs, however, the theoretical lightweight does not always mean fast in practice. The parallelism of the multi-branch structure does not reach the theoretical efficiency and the cross-resolution weight computation also slows down the inference of the model. It is even slower than the HRNet during actual running. Benefit from network architecture search (NAS) technology, ViPNAS\cite{vipnas} reaches a notable balance between accuracy and speed, but its performance while handling picture tasks on the CPU is not satisfactory. Other methods are short in terms of parameter quantity\cite{mspn}. Research in the lightweight human pose estimation area is still inadequate.

The SimpleBaseline\cite{simplebaseline} is an elegant and effective method for human pose estimation, they provide the capacity of deconvolution layers in this problem. Based on their successful work, we focus on the design of lightweight and simple network architecture for human pose estimation. First, we use a lightweight network MobileNetV3\cite{mobilenetv3} as the backbone in our model, rather than the huge  ResNet\cite{resnet}. As the latest model in MobileNet series\cite{mobilenetv1,mobilenetv2,mobilenetv3} MobileNetV3 has shown its performance in many computer vision tasks. Then we analyse the parameters and computation cost in the whole model and notice the deconvolution layers contain the dominant resources. Inspired by the design of the MobileNet series, we change the normal deconvolution layers to a depthwise deconvolution layer following a pointwise convolution layer. To compensate for the performance degradation caused by lightweighting, we introduce an attention mechanism. The attention block utilizes channel, spatial and original information of the input feature map to enhance the performance. Besides, we observe that Mean Square Error (MSE) loss is widely used by existing methods during the training stage. MSE loss just calculates the average error for different pixels on the heatmaps, without reflecting the inhomogenous from different locations. We propose a novel loss function called heatmap weighting loss to generate weights for every pixel on heatmaps that is used to calculate the final error.

In short, our contribution can be summarized as followed.
\begin{itemize}
    \item We simplify the deconvolution head network in SimpleBaseline and present an attention mechanism that exploits channel, spatial and global representations to benefit the performance of the lightweight architecture.
    \item We propose a novel loss function - heatmap weighting loss that explores the information from ground truth heatmaps to improve keypoints localization.
\end{itemize}

\section{Related Work}
\subsection{Human Pose Estimation}
Human posture estimation has been an active research problem in the last decades. Before the rise of deep learning, human pose estimation was mainly based on graph structure models, using picture global features or handcraft filters to detect and localize the keypoints of human body\cite{traditional}. Since Toshev et al.\cite{deeppose} propose DeepPose, which uses the deep neural network to regress the keypoints locations, the neural network based approach becomes mainstream. Newell et al.\cite{hourglass} propose Hourglass, which consists of multi-stage stacked hourglass networks, it is a milestone on the MPII\cite{mpii} benchmark dataset. Chen et al.\cite{cpn} propose Cascaded Pyramid Network (CPN), which is the champion of COCO 2017 keypoint challenge. CPN uses a GlobalNet\cite{cpn} to estimate simple keypoints and handle complex keypoints with a RefineNet\cite{cpn}. Xiao et al.\cite{simplebaseline} propose SimpleBaseline, which is a simple but effective method composed of a backbone and several deconvolution layers. Sun et al.\cite{hrnet} present a novel High-Resolution Net to maintain high-resolution representation by connecting multi-resolution convolutions in parallel and conducting repeated multi-scale feature map fusion. Cai et al.\cite{prm} introduce Residual Steps Network (RSN) to learn more delicate local feature by intra-level information fusion, and propose Pose Refine Machine (PRM) to refine feature map by attention mechanism. This method is the winner of COCO 2019 keypoint challenge.

\subsection{Lightweight Network Design}
Design deep neural network architecture for achieving a trade-off between efficiency and accuracy has been an active research topic, especially in industrial research. In recent years, a large number of compact network architectures have been proposed. Xception\cite{xception} uses depthwise separable convolution operation to reduce parameters and computation. MobileNets series\cite{mobilenetv1, mobilenetv2, mobilenetv3} are also based on depthwise convolution and pointwise convolution. MobileNetV2\cite{mobilenetv2} present inverted residual block and MobileNetV3\cite{mobilenetv3} exploit NAS method searching a optimal structure that has better performance and fewer FLOPs. ShuffleNet\cite{shufflenet} proposes a channel shuffle operation for information interaction between feature map groups. ShuffleNetV2\cite{shufflenetv2} studies the lightweight network design principles and modifies the ShuffleNet according to the proposed guidelines. GhostNet\cite{ghostnet} investigates the correlation and redundancy between feature maps and presents a cheap operation to reduce the computation of pointwise convolution. The development of lightweight networks has great significance for our work, on the one hand, we can directly use a more powerful lightweight network as a backbone in our model, on the other hand, we can draw on the idea of compact network design in our downstream tasks.

\subsection{Attention Mechanism}
Attention mechanism has shown powerful performance in many computer vision tasks such as image classification, scene segmentation, objection detection, and optical character recognition (OCR). Wang et al.\cite{nonlocal} propose Non-Local Networks which introduce self-attention to capture long-range dependencies. Hu et al.\cite{se} present Squeeze-and-Excitation Networks (SENet), using weight learned from global information to excite or suppress different channels, which is the champion of ILSVRC 2017. Li et al.\cite{sknet} expand SENet and propose Selective Kernel Networks (SKNet), using feature maps generated by multiple kernel sizes to gather attention value. Woo et al.\cite{cbam} propose Convolutional Block Attention Module (CBAM) to exploit both channel and spatial information to refine the feature map. Most attention modules can be easily embedded in our pose estimation network without introducing too much extra computation while improving the performance of the network.

\section{Method}
\subsection{Lightweight Deconvolution Head Network}
Deconvolution head network is a different method to rehabilitate high-resolution feature map which is introduced in SimpleBaseline. Before that, upsampling is the dominant technology to recover high-resolution feature map\cite{hourglass, cpn}, and some convolution layers added to adjust the generated feature map. In the deconvolution head network, a deconvolution layer that could learn the upsampling parameters from input data during training combines upsampling and convolution parameters in a single operation.

The standard deconvolution layer is parameterized by the convolution kernel $K$ of size $D_K\times D_K\times C_{in}\times C_{out}$, where $D_K$ is the spatial dimension of the kernel assumed to be square and $C_{in}$ is the number of input channels, $C_{out}$ is the number of output channels, and $l$ is the number of deconvolution layers. Despite the first layer for channel compression and the last layer for generating output feature map, the total number of parameters in a deconvolution head network is:
\begin{equation}
    \sum_i^l D_{K_i}\times D_{K_i}\times C_{in_i}\times C_{out_i}
\end{equation}

The computational cost of a deconvolution layer multiplicatively depends on the size of kernel $D_K \times D_K$, the number of input channels $C_{in}$, the number of output channels $C_{out}$, and the size of feature maps $W\times H$. Therefore, the total computation in a $l$ layer deconvolution head network is:
\begin{equation}
    \sum_i^l D_{K_i}\times D_{K_i}\times C_{in_i}\times C_{out_i} \times W_i \times H_i
\end{equation}
\begin{figure}[!t]
    \centering
    \includegraphics[width=0.9\linewidth]{figure/network.png}
    \caption{Illustration of lightweight deconvolution head network.}
    \label{network}
\end{figure}

Based on the design in MobileNets, we replace the standard deconvolution layer with a depthwise deconvolution layer following a pointwise convolution layer. Fig. \ref{network} illustrates the structure of our lightweight deconvolution head network. Depthwise deconvolution applies a single filter on each channel for upsampling and producing the new representations in high-resolution feature map, while pointwise convolution provides a bridge for information fusion across channels. The influence of the output channels on the parameters and computation is eliminated in depthwise deconvolution, and in pointwise convolution, the kernel size will no longer be a factor. The computation cost of our new deconvolution head network can be written as:
\begin{equation}
    \sum_i^l D_{K_i}\times D_{K_i}\times C_{in_i}\times W_i \times H_i + C_{in_i}\times C_{out_i} \times W_i \times H_i
\end{equation}
The number of parameters is proportional to the computation cost, just remove the size of feature map.
Following the setting in SimpleBaseline, we set the layers to 3, kernel size to 4, and the number of channels to 256 in all layers. Thus we can get a reduction in computation cost of:
\begin{equation}
\begin{aligned}
    &\frac{\sum_i^3 4\times 4\times 256\times W_i \times H_i + 256\times 256 \times W_i \times H_i}{\sum_i^3 4\times 4\times 256\times 256 \times W_i \times H_i} \\
    =&\frac{17}{256}\approx\frac{1}{16}.
\end{aligned}
\end{equation}

\subsection{Attention Mechanism}
To improve the capacity of our lightweight deconvolution head network, we introduce an attention mechanism. Motivated by the work of SENet, we use an adaptive average pooling to aggregate spatial information in each channel. After generating channel representations, two pointwise convolution layers are used to calculate the correlation between channels. To reduce the parameters, the input feature dimension $C$ is squeezed to $C/r$ after the first convolution, where $r$ is the squeeze ratio. Before finally broadcasting along spatial dimension and multiplying with the original feature map, a hard sigmoid function is utilized to normalize the weight. Define the feature map as $f$, a convolution layer with $p$ output channels and kernel size of $m\times n$ as $\text{F}^{m\times n}_{p}$, the hard sigmoid as $\sigma(\cdot)$, the global average pooling as $\text{GAP}(\cdot)$, and the element-wise multiplication as $\odot$. The channel attention $\mathcal{H}_{channel}(\cdot)$ is computed as:
\begin{equation}
    \mathcal{H}_{channel}(f) = f \odot \sigma(\text{F}^{1\times1}_{C}(\text{F}^{1\times1}_{C/r}(\text{GAP}(f))))
\end{equation}

\begin{figure}[!t]
    \centering
    \includegraphics[width=0.95\linewidth]{figure/attention.png}
    \caption{Illustration of attention architecture.}
    \label{attention}
\end{figure}

Since human pose estimation is a space-sensitive task, a spatial attention mechanism can help to better mine the information in spatial dimension. To converge information in different channels, we apply a pointwise convolution to get a linearly combined representation. After that, a standard convolution is used to produce the spatial attention map. A hard sigmoid layer is also used to rescale the activation values. Following the definition of the channel portion, the spatial attention $\mathcal{H}_{spatial}(\cdot)$ is written as:
\begin{equation}
    \mathcal{H}_{spatial}(f) = f \odot \sigma(\text{F}^{3\times3}_{1}(\text{F}^{1\times1}_{1}(f)))
\end{equation}

For the arrangement of attention modules, we use a parallel structure, as shown in Fig. \ref{attention}. Channel attention and spatial attention are applied to the feature map individually and an identity connection is also employed. The identity mapping protects local features from being corrupted by dimensional compression in the attention block. Meanwhile, channel attention exploits inter-level representation to benefit keypoints identification and spatial attention improves keypoints by optimizing spatial information. Finally, the feature maps in three branches are added to get the output feature. Our attention mechanism can be formulated as:
\begin{equation}
    f_{out}=f_{in} + \mathcal{H}_{channel}(f_{in}) + \mathcal{H}_{spatial}(f_{in}) \\
\end{equation}

\subsection{Heatmap Weighting Loss}
In the training phase, MSE is commonly used as the loss function\cite{hourglass, simplebaseline, cpn, hrnet}. Beyond the previous work, we argue that the weight of the pixels at different locations on the heatmaps should be various. The loss function should focus more on the points with larger values on the heatmaps, which means devoting more attention to those pixels that are closer to the keypoints. Based on this thinking, we propose heatmap weighting loss which generates weight for each pixel by adopting the transformation function on the ground truth heatmap, as shown in Fig. \ref{loss}. Our loss function presents as:
\begin{equation}
    L=\frac{1}{J}\sum^J_{j=1}\left[\mathcal{F}(P^{GT}_j) + 1\right]\odot\Vert P_j-P^{GT}_j \Vert_2
\end{equation}
where $J$ is the number of keypoints, $P$ and $P^{GT}$ are predicted heatmap and ground truth heatmap respectively, $\mathcal{F}(\cdot)$ is weight generation function, and $\odot$ denotes element-wise multiplication.
\begin{figure}[!t]
    \centering
    \includegraphics[width=\linewidth]{figure/loss.png}
    \caption{Proposed heatmap weighting loss.}
    \label{loss}
\end{figure}

For the selection of weight generation function, in consideration of the target heatmaps are usually sampled from a two-dimensional Gaussian distribution\cite{heatmap}. We select several convex functions on $[0,1]$ interval such as linear function, power function, and exponential function to generate the weights and evaluate their performance. See Section \ref{subsec:loss} for details.

\section{Experiments}
We evaluate our method on COCO\cite{coco} and MPII\cite{mpii} datasets. Following the two-stage top-down paradigm\cite{simplebaseline, hrnet}, our approach estimates keypoints confidence maps from the detected person boundary box. We report the comparisons with other methods and also conduct inference speed test and ablation studies on these datasets. The project page \footnote{\url{https://github.com/HAIV-Lab/ICPR22w}} has been set up with programs available.

\subsection{Experimental Setup}
\subsubsection{Dataset and metric}
COCO dataset contains over 200K images and 250K person instances with 17 keypoints. Our model is trained on the train2017 set (includes 57K images and 150K person instances) and validated on the val2017 set (includes 5K images). MPII includes around 25K images containing over 40K people with annotated body joints. We adopt object keypoint similarity (OKS) based mean average precision (AP) to evaluate the performance on COCO, for MPII, the standard PCKh (head-normalized percentage of correct keypoints) is used.

\subsubsection{Image cropping}
The human box is expanded in height or width to reach a fixed aspect ratio, e.g., height:width=4:3. Then we crop the box area from the image and finally resize it to a certain resolution such as $256\times192$ for COCO and $256\times256$ for MPII.

\subsubsection{Data augmentation}
We apply a series of data augmentation operations to make the model learn scale invariance and rotation invariance. Augmentation strategy includes flip, random rotation ($[-80^\circ, 80^\circ]$), random scale ([0.5, 1.5]), and half body augmentation which keeps only the upper or the lower body at random for COCO.

\subsubsection{Training}
We use Adam optimizer mini-batch size 64 to train our network, the initial learning rate is 5e-4 and reduced by a factor of 10 at the 170th and 200th epoch. The training process is terminated within 210 epochs. The warm-up strategy with a liner warm ratio of 0.001 is used in  the beginning 500 iterators\cite{mmpose2020}. Our experiments are conducted on a single NVIDIA 3090 GPU.

\subsubsection{Testing}
In order to minimize the variance of prediction, following the normal practice\cite{hourglass}, we average the heatmaps predicted from the original and flipped images. A quarter of a pixel offset in the direction from the highest response to the second-highest response is applied before the predicted keypoint locations transform back to the original coordinate space of the input image.

\subsection{Results}

\subsubsection{COCO val}
\begin{table*}[!t]
\renewcommand{\arraystretch}{1.5}
\caption{Comparison on COCO val2017 set.}
\label{cocoval}
\centering
\begin{tabular}{llccccccccc}
\hline
\multicolumn{1}{c}{Method} & \multicolumn{1}{c}{Backbone} & Input Size & \#Params & GFLOPs & AP & AP$^{50}$ & AP$^{75}$ & AP$^M$ & AP$^L$ & AR \\
\hline
SimpleBaseline\cite{simplebaseline} & MobileNetV2 & $256\times192$ & 9.6M & 1.59 & 64.6 & 87.4 & 72.3 & 61.1 & 71.2 & 70.7 \\
SimpleBaseline & MobileNetV2 & $384\times288$ & 9.6M & 3.57 & 67.3 & 87.9 & 74.3 & 62.8 & 74.7 & 72.9 \\
SimpleBaseline & MobileNetV3 & $256\times192$ & 8.7M & 1.47 & 65.9 & 87.8 & 74.1 & 62.6 & 72.2 & 72.1 \\
SimpleBaseline & ShuffleNetV2 & $256\times192$ & 7.6M & 1.37 & 59.9 & 85.4 & 66.3 & 56.5 & 66.2 & 66.4 \\
SimpleBaseline & ShuffleNetV2 & $384\times288$ & 7.6M & 3.08 & 63.6 & 86.5 & 70.5 & 59.5 & 70.7 & 69.7 \\
ViPNAS\cite{vipnas} & MobileNetV3 & $256\times192$ & 2.8M & 0.69 & 67.8 & 87.2 & 76.0 & 64.7 & 74.0 & 75.2 \\
Small HRNet\cite{litehrnet} & HRNet-W16 & $256\times192$ & 1.3M & 0.54 & 55.2 & 83.7 & 62.4 & 52.3 & 61.0 & 62.1 \\
Small HRNet & HRNet-W16 & $384\times288$ & 1.3M & 1.21 & 56.0 & 83.8 & 63.0 & 52.4 & 62.6 & 62.6 \\
Lite-HRNet\cite{litehrnet} & Lite-HRNet-18 & $256\times192$ & 1.1M & 0.20 & 64.8 & 86.7 & 73.0 & 62.1 & 70.5 & 71.2 \\
Lite-HRNet & Lite-HRNet-18 & $384\times288$ & 1.1M & 0.45 & 67.6 & 87.8 & 75.0 & 64.5 & 73.7 & 73.7 \\
Lite-HRNet & Lite-HRNet-30 & $256\times192$ & 1.8M & 0.31 & 67.2 & 88.0 & 75.0 & 64.3 & 73.1 & 73.3 \\
Lite-HRNet & Lite-HRNet-30 & $384\times288$ & 1.8M & 0.70 & 70.4 & 88.7 & 77.7 & 67.5 & 76.3 & 76.2 \\
\hline
Ours & MobileNetV3 & $256\times192$ & 3.1M & 0.58 & 65.8 & 87.7 & 74.1 & 62.6 & 72.4 & 72.1 \\
Ours & MobileNetV3 & $384\times288$ & 3.1M & 1.30 & 69.9 & 88.8 & 77.5 & 66.0 & 76.7 & 75.5 \\
\hline
\end{tabular}
\end{table*}
Table \ref{cocoval} reports the comparison results of our method and other state-of-the-art methods. Our method, trained from input resolution with $256\times192$, achieves a 65.8 AP score. Compared to ShuffleNetV2, our method improves AP by 5.9 points with $42\%$ FLOPs. We use about $35\%$ of the parameters and computational cost while obtaining a gain of 1.2 AP points compared to MobileNetV2. Our method matches the performance of SimpleBaseline using MobileNetV3 as the backbone, with only $35\%$ FLOPs and $39\%$ parameters. Compared to Small HRNet-W16\cite{litehrnet}, we improve 10.6 AP score. Our method also outperforms the Lite-HRNet-18 with 1.0 AP points. There are little gaps (1.3 points and 2.0 points) between our approach and the state-of-the-art methods, Lite-HRNet-30 and ViPNAS, respectively, but our method achieves a much higher inference speed. See Section \ref{subsec:speed} for details. It is worth noting that the goal of this work is to introduce a simple lightweight human pose estimation network with high inference speed rather than improving the score of existing SOTA approache on benchmark datasets. For the input size of $384\times288$, our method achieves 69.9 AP score. Some prediction results are visualized in Fig. \ref{vis}.

\begin{figure*}[!t]
    \centering
    \includegraphics[width=0.95\linewidth]{figure/vis.jpg}
    \caption{Prediction results on COCO val.}
    \label{vis}
\end{figure*}

\subsubsection{COCO test-dev}
Table \ref{cocotest} reports the comparison results of our models and other lightweight methods. Our networks achieve 65.3 and 69.2 with the input size of $256\times192$ and $384\times288$, respectively. Our approach outperforms other methods except for Lite-HRNet-30 in AP scores, and our method is closer to SOTA with larger input size.
\begin{table*}[!t]
\renewcommand{\arraystretch}{1.5}
\caption{Comparison on COCO test-dev2017 set.}
\label{cocotest}
\centering
\begin{tabular}{llccccccccc}
\hline
\multicolumn{1}{c}{Method} & \multicolumn{1}{c}{Backbone} & Input Size & \#Params & GFLOPs & AP & AP$^{50}$ & AP$^{75}$ & AP$^M$ & AP$^L$ & AR \\
\hline
SimpleBaseline\cite{simplebaseline} & MobileNetV2 & $256\times192$ & 9.6M & 1.59 & 64.1 & 89.4 & 71.8 & 60.8 & 69.8 & 70.1 \\
SimpleBaseline & MobileNetV2 & $384\times288$ & 9.6M & 3.57 & 66.8 & 90.0 & 74.0 & 62.6 & 73.3 & 72.3 \\
SimpleBaseline & ShuffleNetV2 & $256\times192$ & 7.6M & 1.37 & 59.5 & 87.4 & 66.0 & 56.6 & 64.7 & 66.0 \\
SimpleBaseline & ShuffleNetV2 & $384\times288$ & 7.6M & 3.08 & 62.9 & 88.5 & 69.4 & 58.9 & 69.3 & 68.9 \\
Small HRNet\cite{litehrnet} & HRNet-W16 & $384\times288$ & 1.3M & 1.21 & 55.2 & 85.8 & 61.4 & 51.7 & 61.2 & 61.5 \\
Lite-HRNet\cite{litehrnet} & Lite-HRNet-18 & $256\times192$ & 1.1M & 0.20 & 63.7 & 88.6 & 71.1 & 61.1 & 68.6 & 69.7 \\
Lite-HRNet & Lite-HRNet-18 & $384\times288$ & 1.1M & 0.45 & 66.9 & 89.4 & 74.4 & 64.0 & 72.2 & 72.6 \\
Lite-HRNet & Lite-HRNet-30 & $256\times192$ & 1.8M & 0.31 & 66.7 & 88.9 & 74.9 & 63.9 & 71.9 & 72.7 \\
Lite-HRNet & Lite-HRNet-30 & $384\times288$ & 1.8M & 0.70 & 69.7 & 90.7 & 77.5 & 66.9 & 75.0 & 75.4 \\
\hline
Ours & MobileNetV3 & $256\times192$ & 3.1M & 0.58 & 65.3 & 89.7 & 73.4 & 62.6 & 70.4 & 71.3 \\
Ours & MobileNetV3 & $384\times288$ & 3.1M & 1.30 & 69.2 & 90.6 & 76.9 & 65.8 & 74.9 & 74.7 \\
\hline
\end{tabular}
\end{table*}

\subsubsection{MPII val}
Comprehensive results on MPII are reported in Table 3. Our method achieves 85.9 PCKh with a standard $256\times256$ input size, outperforms the ShuffleNetV2, MobileNetV2 by 3.1 and 0.5 points, respectively.
\begin{table*}[!t]
\renewcommand{\arraystretch}{1.5}
\caption{Comparison on MPII val set.}
\label{mpii}
\centering
\begin{tabular}{llcccccccccc}
\hline
\multicolumn{1}{c}{Method} & \multicolumn{1}{c}{Backbone} & \#Params & GFLOPs & Head & Shoulder & Elbow & Wrist & Hip & Knee & Ankle & PCKh \\
\hline
SimpleBaseline\cite{simplebaseline} & MobileNetV2 & 9.6M & 2.12 & 95.3 & 93.5 & 85.8 & 78.5 & 85.9 & 79.3 & 74.4 & 85.4 \\
SimpleBaseline & ShuffleNetV2 & 7.6M & 1.83  & 94.6 & 92.4 & 83.0 & 75.6 & 82.8 & 75.9 & 69.2 & 82.8 \\
Lite-HRNet\cite{litehrnet} & Lite-HRNet-18 & 1.1M & 0.27 & 96.1 & 93.7 & 85.5 & 79.2 & 87.0 & 80.0 & 75.1 & 85.9 \\
Lite-HRNet & Lite-HRNet-30 & 1.8M & 0.42 & 96.3 & 94.7 & 87.0 & 80.6 & 87.1 & 82.0 & 77.0 & 87.0 \\
\hline
Ours & MobileNetV3 & 3.1M & 0.77 & 95.6 & 93.9 & 85.1 & 79.5 & 86.3 & 80.4 & 75.5 & 85.9 \\
\hline
\end{tabular}
\end{table*}

\subsection{Weight Generation Function}
\label{subsec:loss}
We experimentally study the influence of heatmap weighting loss. We evaluate different weight generation functions and compare them with the traditional MSE loss. Table \ref{lossablation} reports the results on COCO val with the input size of $256\times192$.
\begin{table}[!t]
\renewcommand{\arraystretch}{1.5}
\caption{Comparison of different weight generation functions.}
\label{lossablation}
\centering
\begin{tabular}{ccccc}
\hline
Weight Generation Function & AP & AP$^{50}$ & AP$^{75}$ & AR \\
\hline
None & 65.56 & 87.36 & 73.97 & 71.65 \\
$\mathcal{F}(x)=x$ & 65.83 & 87.70 & 74.06 & 72.06 \\
$\mathcal{F}(x)=2x$ & 65.59 & 87.37 & 74.01 & 71.90 \\
$\mathcal{F}(x)=x^2$ & 65.65 & 87.70 & 73.96 & 71.81 \\
$\mathcal{F}(x)=e^x$ & 65.70 & 87.66 & 73.74 & 71.79 \\
\hline
\end{tabular}
\end{table}

We can find that the AP score got the most significant gain of 0.27 points when using the most simple endomorphism as the weight generation function. When we increase the transformation ratio in linear mapping, the gain becomes negligible (only 0.03 AP score). We also attempt the square and exponential functions that obtain gains of 0.09 and 0.14 points, respectively. Although the gain from our heatmap weighting loss function is not significant now, it provides a new optimization idea for human pose estimation. The notion of our heatmap weighting loss can be considered as a kind of adjustment of the handcrafted two-dimensional Gaussian distribution heatmap to make it approximate the actual distribution of the human keypoint.

\subsection{Inference Speed}
\label{subsec:speed}
Operations (OPs) denote the number of computations required for the forward inference of the model and reflect the demand for hardware computational units. While float is the most commonly used data type, FLOPs becomes the general reference for evaluating model size. The actual inference speed on hardware is a complex issue. It is not only affected by the amount of computation, but also by many factors such as the amount of memory access, hardware characteristics, and system environment. For this reason, we study the actual inference speed of different human pose estimation models on an edge device.

Our test platform is a laptop equipped with Intel Core i7-10750H CPU and NVIDIA GTX 1650Ti (Notebooks) GPU. We perform 50 rounds of inference with each model and calculate the average iterations per second. To reduce the error, the initial 5 iterations are not included in the statistics. The time for data pre-processing is eliminated and we keep the same input feature map from COCO val for each model with $256\times192$ resolution. We conduct several sets of experiments and calculate the average FPS as the final result. Table \ref{speed} and Fig. \ref{speed-pic} summarize the results of our speed experiment.

Our experiments confirm that FLOPs do not fully respond to inference speed. The GFLOPs of Lite-HRNet-18 and Lite-HRNet-30 are impressive 0.20 and 0.31, respectively. However, dragged down by the multi-branch parallel convolution architecture, the lite-HRNet series only achieves about 10 FPS on both CPU and GPU platforms. Their inference speed on GPU device is even slower than the speed of MobileNetV2, ShuffleNetV2, and our method on CPU platform. ViPNAS, the most accurate method, reaches 22.6 FPS on the GPU, but less than 5 FPS on the CPU. Our method achieves 55.3 FPS on GPU, which is 7.4 times higher than Lite-HRNet-30 and 2.4 times higher than ViPNAS. On the CPU platform, the FPS of our method is 18.3, about 4 times higher than that of ViPNAS. Our approach achieves higher accuracy while being essentially the same speed as MobileNetV2 and ShuffleNetV2. Inference speed experiments demonstrate that our method is more friendly to the mobile device and better suited to real-world applications.

\begin{table}[!t]
\renewcommand{\arraystretch}{1.5}
\caption{comprarison of inference speed.}
\label{speed}
\centering
\begin{tabular}{lcccc}
\hline
\multicolumn{1}{c}{Model} & AP & FPS(GPU) & FPS(CPU) & GFLOPs \\ \hline
MobileNetV2\cite{mobilenetv2} & 64.6 & 57.6 & 19.3 & 1.59\\
ShuffleNetV2\cite{shufflenetv2} & 59.9 & 51.8 & 20.2 & 1.37\\
ViPNAS\cite{vipnas} & 67.8 & 22.6 & 4.6 & 0.69\\
Lite-HRNet18\cite{litehrnet} & 64.8 & 12.8 & 10.3 & 0.20\\
Lite-HRNet30\cite{litehrnet} & 67.2 & 7.5 & 6.2 & 0.31\\ \hline
Ours & 65.8 & 55.2 & 18.3 & 0.58\\
\hline
\end{tabular}
\end{table}

\begin{figure}[!t]
    \centering
    \includegraphics[width=\linewidth]{figure/speed.png}
    \caption{Speed-accuracy trade-off on COCO val. The circular area represents the number of parameters of the corresponding model.}
    \label{speed-pic}
\end{figure}

\subsection{Ablation Study}
\label{subsec:ablation}
In this part, we validate the improvement of our attention architecture by performing ablation experiments on COCO val set with $256\times192$ input size, as shown in Table 7.   We compare the prior attention mechanism SE-Block, CBAM, and our attention module.

\begin{table}[!t]
\renewcommand{\arraystretch}{1.5}
\caption{Comparison of different attention blocks.}
\label{attentionablation}
\centering
\begin{tabular}{ccccc}
\hline
Attention & AP & AP$^{50}$ & AP$^{75}$ & AR \\
\hline
None & 65.29 & 87.44 & 73.28 & 71.53 \\
SE-Block\cite{se} & 65.71 & 87.57 & 74.10 & 71.93 \\
CBAM\cite{cbam} & 65.77 & 87.92 & 74.19 & 71.02 \\
\hline
Ours & 65.83 & 87.70 & 74.06 & 72.06 \\
\hline
\end{tabular}
\end{table}
Our approach outperforms the other two modules in AP score, which implies trimming the common attention block designed for image classification before applying it to a certain task is necessary.

\section{Conclusion}
In this paper, we propose a lightweight human pose estimation network with high inference speed by simplifying the deconvolution head network in SimpleBaseline and applying an attention mechanism that exploits channel, spatial and global representations to improve the performance. We also present a novel loss function, heatmap weighting loss, which explores the information from ground truth heatmaps to benefit keypoints localization. Our method achieves a balance between performance, resource volume, and inference speed. We hope that this work gives a new idea for lightweight human pose estimation.

\newpage
\bibliographystyle{IEEEtran}


\section{Introduction}
\label{introduction}

The Unified  Medical  Language  System  (UMLS) Metathesaurus is a biomedical terminology integration system developed by the US National Library of Medicine to integrate biomedical terms by organizing clusters of synonymous terms into concepts. The current construction process of the UMLS Metathesaurus heavily relies on lexical similarity algorithms to identify candidates for synonymy, although terms that do not share a common semantics are prevented from being recognized as synonymous. Additionally, synonymy asserted between terms in a source vocabulary is generally preserved in the Metathesaurus. Final synonymy determination comes from human curation by the Metathesaurus editors. Given the current size of the Metathesaurus with over 15 million terms from 214 source vocabularies, this process is inevitably costly and error-prone (as pointed out by \cite{cimino1998auditing,cimino2003consistency,jimeno2009reuse,morrey2009neighborhood,mougin2009analyzing}.)

\textbf{Motivation.}
Recent successes in the field of natural language processing as well as the enormous knowledge accumulated over 30 years of manual curation has provided us ample material to tackle the problem of reconstructing the UMLS Metathesaurus using supervised learning approaches. We consider the reconstructing process as a large-scale synonymy prediction problem. The full scale of the this problem (e.g., $8.7M^2$ in the UMLS 2020AA version) is extremely large and impractical (even when restricted to English terms from active vocabularies), since $8.7M$ biomedical terms need to be compared pairwise. Therefore, our strategy for addressing this problem is finding the best performing approaches in the sampling evaluations as candidates for later full scale evaluations. We have recently achieved preliminary results \cite{bajaj2021evaluating,nguyen2021adding,nguyen2022conlm,nguyen2021biomedical,wijesiriwardene2022ubert} in the sampling evaluation phase.

In our recent work \cite{nguyen2021biomedical}, we formalized synonymy prediction in the UMLS Metathesaurus as a vocabulary alignment problem, referred to as the UMLS Vocabulary Alignment (UVA) problem. Given a pair of biomedical terms from the UMLS source vocabularies, we developed different approaches for predicting if the terms are synonymous. For evaluating any approach for the UVA task, we generated a set of dataset variants (also referred to as ``seed alignment'' or ``ground truth'') with different degrees of lexical similarity among the negative examples as described in \cite{nguyen2021biomedical}. These UVA datasets containing hundreds of millions of pairs of biomedical terms from the active subset of source vocabularies in the UMLS 2020AA have been used for evaluating the approaches as follows.

Our recent work \cite{bajaj2021evaluating,nguyen2021adding,nguyen2022conlm,nguyen2021biomedical,wijesiriwardene2022ubert} developed several approaches for addressing the UVA problem. In \cite{nguyen2021biomedical}, we described the RBA (rule-based approximation) approach that approximates the current construction process into a set of rules. In \cite{nguyen2021biomedical}, we also described LexLM, a supervised learning approach that utilized the Siamese architecture with LSTM and BioWordVec \cite{zhang2019biowordvec} embeddings. Although the LexLM largely outperformed the RBA, we noted as a limitation of this work that we only leveraged lexical information and did not include any contextual information. In \cite{nguyen2022conlm}, we addressed this limitation of the LexLM model by incorporating the contextual information into the ConLM model. We also attempted to improve the performance of the LexLM by adding an attention layer into the neural network in \cite{nguyen2021adding}. In \cite{bajaj2021evaluating} we evaluated the role of different biomedical embeddings such as BioBERT \cite{lee2020biobert} and SapBERT \cite{liu2020self} compared to the BioWordVec in the LexLMs. Additionally, in \cite{wijesiriwardene2022ubert} we fine-tuned the BERT models such as BioBERT and SapBERT for the UVA task.

\textbf{Shared resources.}
We believe that sharing our recent work as reusable and reproducible resources will benefit broader research communities in many different directions. Besides using our work in the internal applications at the National Library of Medicine, we are interested in engaging and supporting external research communities. While exploring several techniques as described above, given the large scale and domain diversity of the UVA datasets compared to the related work, we believe that the UVA problem and its extended biomedical applications will offer various multi-disciplinary challenges for different research communities. Therefore, our goal in this paper is to prepare the materials suitable for the UVA challenge with three specific resources as follows.

\textbf{Contributions.} We share the following resources with broader research communities:
\begin{itemize}
\item A scalable dataset generator taking input files from any UMLS version and generate UVA datasets with customizable parameters (Section \ref{generator}). The new UMLS releases will be used for generating the new UVA challenge datasets.
\item A set of UVA datasets from the UVA generator, particularly for the UMLS Metathesaurus 2020AA, 2021AA and 2021AB  (Section \ref{generating_datasets}).
\item A set of UVA baseline approaches (Section \ref{baselines}) and reproducible experiments including RBA, LexLM, and ConLM (Section \ref{experiments}).
\end{itemize}

These resources are connected in a multi-step pipeline from reading the input files from a particular UMLS Metathesaurus,  generating the datasets, and running the experiments with the baseline approaches. All the UVA resources including the source code, the datasets, and the instructions for running the experiments are available at \url{https://w3id.org/uva}.

The remainder of the paper is organized as follows. Section \ref{background} provides background knowledge in the Metathesaurus. Section \ref{generator} provides the key information about the dataset generator. Section \ref{baselines} summarizes the three baseline approaches for the UVA task. In Section \ref{experiments}, we present our experiments for the three baselines and their results. Section \ref{discussion} discusses our future work and the maintenance plan for the UVA resources. In section \ref{related}, we discuss the related work. Section \ref{conclusion} concludes the paper.

\section{Knowledge Representation in the UMLS Metathesaurus for the UVA Problem}
\label{background}

This section summarizes the key concepts in the UMLS Metathesaurus to be used in the UVA problem. Additional details and examples can be found in \cite{nguyen2022conlm,nguyen2021biomedical} and the UMLS manual.

\subsection{Key Concepts in the UMLS Metathesaurus}
Let $T$ = ($S_{STR}$, $S_{SRC}$, $S_{SCUI}$, $S_{SG}$) be the set of all input tuples in the Metathesaurus where $S_{STR}$ is the set of all strings, $S_{SRC}$ is the set of all sources, $S_{SCUI}$ is the set of all source concept unique identifiers, and $S_{SG}$ is the set of all semantic groups.

\textbf{AUI and $m_a$ link mapping function.} Every occurrence of a term in a source vocabulary is assigned a unique atom identifier (AUI).
Let $S_{AUI}$ be the set of all AUIs in the Metathesaurus. Let $m_a$ be the function that maps concept string $str \in S_{STR}$ from source vocabulary $src \in S_{SRC}$ to a new AUI $a \in S_{AUI}$ such that $a$ = $m_a(str, src)$.

\textbf{SUI and $m_s$.} These AUIs are then linked to a unique string identifier (SUI) to represent occurrences of the same string. Any lexical variation in character set, upper-lower case, or punctuation will result in a separate SUI.
Let $S_{SUI}$ be the set of all SUIs in the Metathesaurus. Let $m_s$ be the function that maps an AUI $a \in S_{AUI}$ to a new SUI $s \in S_{SUI}$ such that $s$ = $m_s(a)$.

\textbf{LUI and $m_l$.}
All the English lexical variants of a given string (detected using the Lexical Variant Generator tool \cite{mccray1994lexical}) are associated with a single normalized term (LUI).
Let $S_{LUI}$ be the set of all LUIs in the Metathesaurus. Let $m_l$ be the function that maps a SUI $s \in S_{SUI}$ to a new LUI $l \in S_{LUI}$ such that $l$ = $m_l(s)$.

\textbf{SCUI and $m_s$}. Each AUI is optionally associated with one identifier provided by its source (``Source CUI'' or SCUI).
Let $S_{SCUI}$ be the set of all SCUIs in the Metathesaurus. Let $m_u$ be the function that maps a concept string $a \in S_{AUI}$ to a new SCUI $u \in S_{SCUI}$ such that $u$ = $m_u(a)$.

\textbf{CUI.} Lexical similarity forms the basis for suggesting synonymy in the UMLS Metathesaurus.
Let $S_{CUI}$ be the set of all concepts CUIs.

\textbf{Source Semantic Group and $m_g$.}  We obtain the semantic group from the semantic type of the concept CUI. Let $S_{SG}$ be the set of semantic groups in the Metathesaurus.
Let $m_g$ be the function that maps an SCUI $s \in S_{SCUI}$ to a set of semantic groups such that $m_g(s) \subset S_{SG}$.

\textbf{Source Hierarchical Relations and $m_h$.} An SCUI may have parent or child terms in a source vocabulary.
Let $m_h$ be the function that maps an SCUI $s \in S_{SCUI}$ to a set of its parents, $m_h: S_{SCUI} \rightarrow S_{SCUI}$ such that $m_h(s) \subset S_{SCUI}$.

We will use these key concepts in the remaining sections.

\subsection{The UVA Problem}
The synonymy prediction task is defined in \cite{nguyen2021biomedical} as follows.

$T$ is the set of all input tuples ($S_{STR}$, $S_{SRC}$, $S_{SCUI}$, $S_{SG}$) from source vocabularies.
Let $t$ = ($str$, $src$, $scui$, $sg$) $\in$ $T$, and $t'$ = ($str'$, $src'$, $scui'$, $sg'$) $\in$ $T$.
Let $p$: $T$ $\times$ $T$ $\rightarrow$ \{0,1\} be the prediction function mapping a pair of input tuples to either 0 or 1.
The two strings $str$ from $t$ and $str'$ from $t'$ are synonymous if $p$($t$,$t'$) = 1.

\section{UVA Dataset Generator}
This section summarizes the key points about the UVA data generator implemented in \cite{nguyen2021biomedical} and the datasets generated by this generator. Additional details can be found in this referenced paper.

\label{generator}
\textbf{Ground truth.} Labeled data are taken from the pairs of atoms that are linked to the same (positive) or different (negative) concepts.
Let $POS$ be the set of positive pairs and $NEG$ be the set of negative pairs.

Given a pair of tuples $t$ = ($str$, $src$, $scui$, $sg$) and $t'$ = ($str'$, $src'$, $scui'$, $sg'$), $a$ = $m_a$($str$, $src$), $a'$ = $m_a$($str'$, $src'$), let $m_c$ be the mapping function respectively linking $a, a' \in S_{AUI}$ to $c, c' \in S_{CUI}$ such that $c$ = $m_c$($a$) and $c'$ = $m_c$($a'$), if $c$ = $c'$ then ($a$, $a'$) $\in POS$ else ($a$, $a'$) $\in NEG$.

\textbf{Data generation principles.} Two principles are used to generate the experimental datasets: (1) provide different degrees of lexical similarity in the negative pairs, and (2) maximize the coverage of AUIs in the training datasets.

Jaccard index is used as a measure for the similarity between atoms. To ignore minor variation among atoms (e.g., singular/plural differences), the lexical similarity of normalized strings is used rather than original strings.
Let $norm$ be the normalizing function that maps a $sui$ to its normalized string, and $m_s$ be the function mapping an AUI to its SUI. The JACC score assessing the similarity between two AUIs is computed as follows.
\begin{equation}
JACC(a, a') = \dfrac{| norm(m_s(a))\cap norm(m_s(a'))|}{|norm(m_s(a)) \cup norm(m_s(a'))|}
\end{equation}

\textbf{Degrees of similarity in negative pairs.} All of the negative pairs in the Metathesaurus are divided into two mutually exclusive sets: (1) SIM, the negative pairs with some similarity ($JACC > 0$) between the two atoms, and (2) NOSIM, the negative pairs that have no similarity ($JACC = 0$) between the two atoms.

Given a pair of tuples $t$ = ($str$, $src$, $scui$, $sg$) and $t'$ = ($str'$, $src'$, $scui'$, $sg'$), with $a$ = $m_a$($str$, $src$), $a'$ = $m_a$($str'$, $src'$), and ($str$, $str'$) $\in NEG$, if $JACC(a, a') > 0$, then ($a, a'$) $\in$ SIM, else ($a, a'$) $\in$ NOSIM.

\textbf{Variants of the negative dataset.} Using the two principles described above, four variants of the negative dataset are created as follows.\\
$NEG_{TOPN}(SIM)$: negative pairs with the highest similarity scores.\\
$NEG_{RAN}(SIM)$: random negative pairs having some similarity.\\
$NEG_{RAN}(NOSIM)$: random negative pairs having no similarity.\\
$NEG_{ALL}$ = $NEG_{TOPN}(SIM)$ $\cup$ $NEG_{RAN}(SIM)$ $\cup$ $NEG_{RAN}(NOSIM)$: include all of the above pairs.

Formally, the number of positive and negative pairs in each dataset variant is computed as follows.
Let $m_c$ be the ground truth function mapping an AUI $a$ to its concept CUI $c$, $c$ = $m_c(a)$.
Let $m_{ca}$ be the function mapping a CUI $c$ to its AUIs $a$, then $m_{ca}(c)$ = \{$a: c = m_c(a)$\}.
Let $n$ be the number of AUIs within a CUI, then $n(a) = |m_{ca}(m_c(a))|$ = |\{$a': c = m_c(a')$\}|.
Let ($a, a'$) be an ordered pair of AUIs, then for every AUI $a$ having $k = (n(a) - 1)$ positive pairs (a,*).
$NEG_{TOPN}(SIM)$ includes $2*k$ negative pairs (a,*) or only 1 negative pair if k = 0.
$NEG_{RAN}(SIM)$ includes $2*k$ negative pairs (a,*) or only 1 negative pair if k = 0.
$NEG_{RAN}(NOSIM)$ includes $2*k$ negative pairs (a,*).
$NEG_{ALL}$ includes up to $6*k$ negative pairs (a,*).

Two types of datasets: (1) learning datasets for training and validating the neural network models, and (2) generalization datasets for testing the generalization of the neural network models. The datasets of the two types are mutually exclusive.

Four dataset variants (TOPN\_SIM, RAN\_SIM, RAN\_NOSIM, and ALL) are created for each dataset type. The set of positive pairs, POS, is split randomly into the learning and generalization datasets (80:20 ratio). The positive learning datasets (80\% of POS) will be combined with the one half of the negative dataset for a given variant. Similarly, the positive generalization datasets (20\% of POS) will be combined with other half of the negative datasets for a given variant.

\section{UVA Baselines}
\label{baselines}

This section summarizes the main points from the three existing baselines from our prior work including the rule-based approximation, the LexLMs \cite{nguyen2021biomedical}, and the ConLMs \cite{nguyen2022conlm}. Additional details can be found in these original papers.

\subsection{Rule-based Approximation (RBA)}

Our RBA baseline \cite{nguyen2021biomedical} uses a set of editorial rules that approximates the current Metathesaurus construction process.

\textbf{Source synonymy (SS) rule.} The two input tuples are synonymous if they have the same identifier in a given source (SCUI). Formally, given a tuple pair $t$ = ($str$, $src$, $scui$, $sg$) $\in T$ and $t'$ = ($str'$, $src'$, $scui'$, $sg'$) $\in T$, let $p_{ss}$ be the prediction function for the source synonymy rule: if $scui$ = $scui'$ then $p_{ss}$($t$, $t'$) = 1.

\textbf{Lexical similarity and semantic compatibility (LS\_SC) rule.} Given a tuple pair $t$ = ($str$, $src$, $scui$, $sg$) $\in T$ and $t'$ = ($str'$, $src'$, $scui'$, $sg'$) $\in T$, let $p_{lssc}$ be the prediction function for the lexical and semantic similarity rule: $p_{lssc}$($t$,$t'$) = 1 if \\
(1) $aui$ = $m_a(str, src)$, $sui$ = $m_s(aui)$, $lui$ = $m_l(sui)$, $sg$ = $m_g(aui)$,               (deriving lui and sg)
$aui'$ = $m_a(str', src')$, $sui'$ = $m_s(aui')$, $lui'$ = $m_l(sui')$, $sg'$ = $m_g(aui')$,\\
(2) $lui$ = $lui'$ and $sg$ $\cap$ $sg'$ $\neq$ $\emptyset$ (asserting lui and sg).

 \textbf{Rule combination (SS\_LS\_SC)}.
Given a tuple pair $t$ = ($str$, $src$, $scui$, $sg$) $\in T$ and $t'$ = ($str'$, $src'$, $scui'$, $sg'$) $\in T$, let $p_{sslssc}$ be the prediction function for the source synonymy and the lexical and semantic similarity rule: $p_{sslssc}$($t$,$t'$) = 1 if $p\_{ss}$($t$, $t'$) = 1 or $p\_{lssc}$($t$, $t'$) = 1.

  \textbf{Transitivity.} The combination rule $SS\_LS\_SC$ can be further amplified by considering its transitive closure.
  Given $t_1, t_2, t_3$ $\in T$, let $p_{trans}$ be the prediction function for the transitivity rule: $P$ = \{$p_{ss}$, $p_{lssc}$, $p_{sslssc}$, $p_{trans}$\} is the set of prediction functions, $p_{trans}$($t_1$, $t_3$) = 1 if $\exists$ $p_1, p_2 \in P$ such that $p_1$($t_1$, $t_2$) = 1 and $p_2$($t_2$, $t_3$) = 1.

\begin{figure}[t]
\centering
    \includegraphics[width=.99\textwidth]{figure-nn}
    \caption{The architecture of the neural network in the Context-enriched Learning Model (ConLM) \cite{nguyen2022conlm} created by adding the ConKG embeddings to the Lexical-based Learning Model (LexLM) embeddings (grey boxes) from \cite{nguyen2021biomedical}. The seven KGE technique used for ConKG include TransE, TransR, DistMult, HolE, ComplEx, RESCAL, and ConvKB. }
    \label{fig:nn}
\end{figure}

\subsection{LexLMs}
The LexLM \cite{nguyen2021biomedical} (grey boxes in Figure \ref{fig:nn}) adopts the Siamese architecture \cite{mueller2016siamese} that takes in a pair of inputs and learns representations based on explicit similarity and dissimilarity information defined during training. The inputs (a pair of atoms) are pre-processed and transformed into their numerical representations with BioWordVec embeddings pre-trained from PubMed text corpus and MeSH data \cite{zhang2019biowordvec}. The word embeddings are then fed to Long Short Term Memory (LSTM) layers to learn the semantic and syntactic features of the atoms through time. The LexLM relies exclusively on the lexical features of the atoms, i.e., the terms themselves.

\subsection{ConLMs}
\renewcommand{\arraystretch}{1}
\begin{table}
\centering
\caption{Set of ConKG embeddings for each ConKG variant}
\label{tab:context_derive}
\resizebox{0.99\textwidth}{!}{%
\begin{tabular}{@{}|c|l|l|@{}}
\toprule
\multicolumn{1}{c|}{\textbf{Variant}} &
\multicolumn{1}{c|}{\textbf{ConKG Triples}} &
\multicolumn{1}{c|}{\textbf{Set of ConKG embedding vectors}} \\ \midrule
$\mathbf{ConSS}$     & $\{(a, r_s, s): s = m_s(a)\}$ & $\{\mathbf{E}_{ConSS}(a) \oplus \mathbf{E}_{ConSS}(m_s(a)): \forall a \in S_{AUI}\}$           \\ \midrule
$\mathbf{ConSG}$    & $\{(s, r_g, g): g \in m_g(s)\}$ & $\{\mathbf{E}_{ConSG}(m_s(a)) \oplus  \mathop{\mathpalette\avsuminner\relax}\displaylimits_{j=1}^{\|m_g(m_s(a))\|} \mathbf{E}_{ConSG}(g_j): \forall a \in S_{AUI}, g_j \in m_g(m_s(a))\}$          \\ \midrule
$\mathbf{ConHR}$   & $\{(s, r_h, p): p \in m_h(s)\}$ & $\{\mathbf{E}_{ConHR}(m_s(a)): \forall a \in S_{AUI}\}$          \\ \midrule
\multirow{3}{*}{$\mathbf{ConAll}$}   & $\{(a, r_s, s): s = m_s(a)\}$  &  $\{\mathbf{E}_{ConAll}(a) \oplus$ \\
 & $\{(s, r_g, g): g \in m_g(s)\}$ &  $\mathbf{E}_{ConAll}(m_s(a)) \oplus$ \\
 & $\{(s, r_h, p): p \in m_h(s)\}$ & $\mathop{\mathpalette\avsuminner\relax}\displaylimits_{j=1}^{\|m_g(m_s(a))\|} \mathbf{E}_{ConAll}(g_j) : \forall a \in S_{AUI}, g_j \in m_g(m_s(a))\}$ \\ \bottomrule
\end{tabular}%
}
\end{table}

The ConLMs \cite{nguyen2022conlm} extends the LexLMs by incorporating the contextual information such as semantic groups, source synonymy, and source hierarchy into the neural networks. The contextual information is represented in the ConKGs as summarized in Table \ref{tab:context_derive}.

$S_{AUI}$ is the set of AUIs, $S_{SCUI}$ is the set of SCUIs, $S_{SG}$ is the set of SGs, and the mapping functions \{$m_s, m_g, m_h$\} are defined in Section \ref{background} above.

\textbf{ConSS.} Let $r_s$ denote the binary relation has\_SCUI from an AUI $a \in S_{AUI}$ to an SCUI $s \in S_{SCUI}$. The ConSS variant includes the triples representing the relationship between an AUI and its SCUI.

\textbf{ConSG.} Let $r_g$ denote the binary relation has\_SG from an SCUI $s \in S_{SCUI}$ to a SG $g \in S_{SG}$. The ConSG variant includes the triples representing the relationship between an SCUI and its semantic groups.

\textbf{ConHR.} Let $r_h$ denote the binary relation has\_parentSCUI from an SCUI $s \in S_{SCUI}$ to its parent SCUI $p \in S_{SCUI}$. The ConHR variant includes the triples representing the relationship between an SCUI and its parent SCUI.

\textbf{ConAll}. ConAll = ConSS $\cup$ ConSG $\cup$ ConHR.

The ConKG triples can be transformed into their knowledge graph embedding vectors using various KG embedding techniques such as TransE \cite{bordes2013translating}, TransR \cite{lin2015learning}, RESCAL \cite{nickel2011three}, DistMult \cite{yang2014embedding}, HolE \cite{nickel2016holographic}, ComplEx \cite{trouillon2016complex}, and ConvKB \cite{nguyen2017novel}

Let $E$ be the set of ConKG entities, $E = S_{AUI} \cup S_{SCUI} \cup S_{SG}$.
Let $R$ be the set of all ConKG relations, $R = \{r_s, r_g, r_h\}$.
Let $\mathbf{T}$ be the set of ConKG triples, a triple $t \in \mathbf{T}$ if $t = (e_1, r, e_2)$ with $r \in R$, and $e_1,e_2 \in E$.
Let $\mathbf{T'}$ be the set of negative ConKG triples, $t' = (e'_1, r, e'_2) \in \mathbf{T'}$ if $\exists t = (e_1, r, e_2) \in \mathbf{T}$ and $t' \notin \mathbf{T} $. Let $d = 2*i$ ($i \in \mathbf{N}$) be the dimension of embedding vector. We choose $d$ to be an even number to facilitate the representation of ComplEx vectors.

\textbf{Entity embeddings.} Let $\mathbf{E}$ be the set of embedding vectors of $E$, then $\mathbf{e} \in \mathbf{E}$ is an embedding vector of entity $e$.
While the embedding for an entity or relation from TransE, HolE, and DistMult is a single vector with dimension $d = 2i$, ComplEx embeddings require two vectors (real and imaginary) of dimension $d = i$. In this case, we concatenate the real and imaginary vectors for each entity into a single vector of dimension $d = 2i$. For ComplEx, we denote the two-vector embeddings as $\textbf{E} = (\textbf{E}_{re}$, $\textbf{E}_{im})$, then we define the embedding vector for an entity $e$ as $\textbf{E}(e) = \textbf{E}_{re}(e) \oplus \textbf{E}_{im}(e)$, $\forall e \in \textbf{E}$.

The output of each KG embedding technique is a set of entity embeddings: $\mathbf{E}_{ConSS}$, $\mathbf{E}_{ConSG}$, $\mathbf{E}_{ConHR}$, and $\mathbf{E}_{ConAll}$, which will be used to derive the ConKG embeddings in Table \ref{tab:context_derive}.

The ConLM \cite{nguyen2022conlm} (Figure \ref{fig:nn}) adds to the LexLM (at the LSTM layer) a specific variant of the ConKGs described in Table \ref{tab:context_derive}. For each ConKG variant, the respective trained ConKG embedding vectors are fed to a 50-unit dense layer to learn the derived features in Table \ref{tab:context_derive}.  The output of the dense layer is then concatenated ($\oplus$)  with the output of the LSTM units from the LexLM. Together, they form the contextualized atom embedding, which is then fed to subsequent dense layers with 128 and 50 learning units, respectively. The output is a Manhattan distance similarity function \cite{aggarwal2001surprising}, which computes a score indicating the degree of synonymy between the atoms with a threshold of 0.5.

\section{Experiments}
\label{experiments}

This section presents the set of experiments for demonstrating the reusability and reproducibility of the UVA baselines described in Section \ref{baselines}. We describe the dataset generation in Section \ref{generating_datasets} and the baseline experiments in Section \ref{generating_baselines}.

\subsection{Experimental Setup}
All these experiments are deployed as batches of parallel jobs with the Slurm~\footnote{https://slurm.schedmd.com/documentation.html} workload manager to the Biowulf high-performance computing cluster~\footnote{https://hpc.nih.gov/} at the National Institutes of Health (NIH).
The \textit{norm} and \textit{gpu} partitions with their corresponding CPU and GPU servers in this cluster (with a limit of 10,000 CPU cores, 60 TB of RAM, and 56 GPUs per user) are used. The following settings are used for deployment to maximize the resources allocated in Biowulf to reduce runtime: (1) multiple nodes (500-625 nodes), (2) multiple threadings with 16-20 threads per node, (3) approximately 95-125 GB of RAM per node.
We used Tesla V100x GPUs with 32GB of GPU RAM and at least 220GB of CPU RAM for each training and testing task.
While the implementation is configurable and reproducible in a different environment, these experiments are computationally- and resource-intensive.
\begin{table}[h]
\centering
\caption{The statistics of UVA datasets generated from UMLS releases 2020AA, 2021AA, and 2021AB \label{statistics}}
\begin{tabular}{@{}|r|r|r|r|@{}}
\toprule
\textbf{Dataset}            & \multicolumn{1}{r|}{\textbf{NEG}} & \multicolumn{1}{r|}{\textbf{POS}} & \multicolumn{1}{r|}{\textbf{Total}} \\ \midrule
20AA\_TRAIN\_ALL                     & 101,322,647                           & 16,743,627                            & 118,066,274                        \\ \midrule
{\color[HTML]{548235} 20AA\_GEN\_ALL}                        & {\color[HTML]{548235} 166,410,710}   & {\color[HTML]{548235} 5,581,208}    & {\color[HTML]{548235} 171,991,918}                        \\ \midrule
{\color[HTML]{305496} 20AA\_GEN\_TOPN\_SIM}                 & {\color[HTML]{305496} 54,752,228}     & {\color[HTML]{305496} 5,581,208}   & {\color[HTML]{305496} 60,333,436}                         \\ \midrule
{\color[HTML]{7030A0} 20AA\_GEN\_RAN\_SIM}                   & {\color[HTML]{7030A0} 54,445,899}    & {\color[HTML]{7030A0} 5,581,208}  & {\color[HTML]{7030A0} 60,027,107}                         \\ \midrule
{\color[HTML]{203764} 20AA\_GEN\_RAN\_NOSIM}                 & {\color[HTML]{203764} 58,256,526}    & {\color[HTML]{203764} 5,581,208}  & {\color[HTML]{203764} 63,837,734}                         \\ \midrule
\textbf{Dataset}                            & \multicolumn{1}{r|}{\textbf{NEG}}  & \multicolumn{1}{r|}{\textbf{POS}} & \multicolumn{1}{r|}{\textbf{Total}} \\ \midrule
21AA\_TRAIN\_ALL                            & 133,165,966                        & 22,614,004                        & 155,779,960                         \\ \midrule
{\color[HTML]{548235} 21AA\_GEN\_ALL}       & {\color[HTML]{548235} 219,495,843} & {\color[HTML]{548235} 7,537,999}  & {\color[HTML]{548235} 227,033,842}  \\ \midrule
{\color[HTML]{305496} 21AA\_GEN\_TOPN\_SIM} & {\color[HTML]{305496} 72,286,689}  & {\color[HTML]{305496} 7,537,999}  & {\color[HTML]{305496} 79,824,688}   \\ \midrule
{\color[HTML]{7030A0} 21AA\_GEN\_RAN\_SIM}  & {\color[HTML]{7030A0} 72,124,757}  & {\color[HTML]{7030A0} 7,537,999}  & {\color[HTML]{7030A0} 79,662,756}   \\ \midrule
21AA\_GEN\_RAN\_NOSIM                       & 76,308,150                         & 7,537,999                         & 83,846,149                          \\ \midrule
\textbf{Dataset}                            & \multicolumn{1}{r|}{\textbf{NEG}}  & \multicolumn{1}{r|}{\textbf{POS}} & \multicolumn{1}{r|}{\textbf{Total}} \\ \midrule
21AB\_TRAIN\_ALL                            & 133,663,761                        & 22,438,339                        & 156,102,100                         \\ \midrule
{\color[HTML]{548235} 21AB\_GEN\_ALL}       & {\color[HTML]{548235} 220,138,372} & {\color[HTML]{548235} 7,479,445}  & {\color[HTML]{548235} 227,617,817}  \\ \midrule
{\color[HTML]{305496} 21AB\_GEN\_TOPN\_SIM} & {\color[HTML]{305496} 72,646,487}  & {\color[HTML]{305496} 7,479,445}  & {\color[HTML]{305496} 80,125,932}   \\ \midrule
{\color[HTML]{7030A0} 21AB\_GEN\_RAN\_SIM}  & {\color[HTML]{7030A0} 72,319,075}  & {\color[HTML]{7030A0} 7,479,445}  & {\color[HTML]{7030A0} 79,798,250}   \\ \midrule
21AB\_GEN\_RAN\_NOSIM                       & 76,413,759                         & 7,479,445                         & 83,893,204                          \\ \bottomrule
\end{tabular}
\end{table}

\subsection{Generating UVA Datasets}
\label{generating_datasets}

\begin{table}[h]
\centering
\caption{Experimental results for the two UVA baselines (ConLM and LexLM) using the 2020AA-ACTIVE datasets with different levels of lexical similarity\label{2020AA-results}}
\begin{tabular}{@{}|rrrrrrrrr|@{}}
\toprule
\multicolumn{9}{|c|}{\textbf{2020AA-ACTIVE}}                                                                                \\ \midrule
\multicolumn{1}{|r|}{}                           & \multicolumn{4}{c|}{{\color[HTML]{548235} \textbf{GEN\_ALL}}}                                                                                & \multicolumn{4}{c|}{{\color[HTML]{305496} \textbf{GEN\_TOPN\_SIM}}}                                                                                \\ \cmidrule(l){2-9}
\multicolumn{1}{|r|}{\multirow{-2}{*}{Baseline}} & \multicolumn{1}{r|}{accuracy}                    & \multicolumn{1}{r|}{precision}                    & \multicolumn{1}{r|}{recall}                       & \multicolumn{1}{r|}{F1}                           & \multicolumn{1}{r|}{accuracy}                    & \multicolumn{1}{r|}{precision}                    & \multicolumn{1}{r|}{recall}                       & F1                           \\ \midrule
\multicolumn{1}{|r|}{RBA}                        & \multicolumn{1}{r|}{{\color[HTML]{548235} 98.63}} & \multicolumn{1}{r|}{{\color[HTML]{548235} 86.31}} & \multicolumn{1}{r|}{{\color[HTML]{548235} 68.71}} & \multicolumn{1}{r|}{{\color[HTML]{548235} 76.51}} & \multicolumn{1}{r|}{{\color[HTML]{2F75B5} 96.14}} & \multicolumn{1}{r|}{{\color[HTML]{2F75B5} 86.83}} & \multicolumn{1}{r|}{{\color[HTML]{2F75B5} 68.71}} & {\color[HTML]{2F75B5} 76.72} \\ \midrule
\multicolumn{1}{|r|}{LexLM}                      & \multicolumn{1}{r|}{{\color[HTML]{548235} 99.38}} & \multicolumn{1}{r|}{{\color[HTML]{548235} 88.75}} & \multicolumn{1}{r|}{{\color[HTML]{548235} 92.54}} & \multicolumn{1}{r|}{{\color[HTML]{548235} 90.61}}  & \multicolumn{1}{r|}{{\color[HTML]{2F75B5} 98.07}} & \multicolumn{1}{r|}{{\color[HTML]{2F75B5} 88.41}} & \multicolumn{1}{r|}{{\color[HTML]{2F75B5} 91.10}} & {\color[HTML]{2F75B5} 89.74} \\ \midrule
\multicolumn{1}{|r|}{ConLM}                      & \multicolumn{1}{r|}{{\color[HTML]{548235} 99.58}} & \multicolumn{1}{r|}{{\color[HTML]{548235} 93.75}} & \multicolumn{1}{r|}{{\color[HTML]{548235} 93.23}} & \multicolumn{1}{r|}{{\color[HTML]{548235} 93.49}} & \multicolumn{1}{r|}{{\color[HTML]{2F75B5} 98.92}} & \multicolumn{1}{r|}{{\color[HTML]{2F75B5} 94.97}} & \multicolumn{1}{r|}{{\color[HTML]{2F75B5} 93.23}} & {\color[HTML]{2F75B5} 94.09} \\ \midrule
\multicolumn{1}{|r|}{}                           & \multicolumn{4}{c|}{{\color[HTML]{7030A0} \textbf{GEN\_RAN\_SIM}}}                                                                                & \multicolumn{4}{c|}{\textbf{GEN\_RAN\_NOSIM}}                                                                                \\ \cmidrule(l){2-9}
\multicolumn{1}{|r|}{\multirow{-2}{*}{Baseline}} & \multicolumn{1}{r|}{accuracy}                    & \multicolumn{1}{r|}{precision}                    & \multicolumn{1}{r|}{recall}                       & \multicolumn{1}{r|}{F1}                           & \multicolumn{1}{r|}{accuracy}                    & \multicolumn{1}{r|}{precision}                    & \multicolumn{1}{r|}{recall}                       & F1                           \\ \midrule
\multicolumn{1}{|r|}{RBA}                        & \multicolumn{1}{r|}{{\color[HTML]{7030A0} 97.02}} & \multicolumn{1}{r|}{{\color[HTML]{7030A0} 98.92}} & \multicolumn{1}{r|}{{\color[HTML]{7030A0} 68.71}} & \multicolumn{1}{r|}{{\color[HTML]{7030A0} 81.09}} & \multicolumn{1}{r|}{{\color[HTML]{203764} 97.26}} & \multicolumn{1}{r|}{{\color[HTML]{203764} 99.99}} & \multicolumn{1}{r|}{{\color[HTML]{203764} 68.71}} & {\color[HTML]{203764} 81.45} \\ \midrule
\multicolumn{1}{|r|}{LexLM}                      & \multicolumn{1}{r|}{{\color[HTML]{7030A0} 99.05}} & \multicolumn{1}{r|}{{\color[HTML]{7030A0} 98.58}} & \multicolumn{1}{r|}{{\color[HTML]{7030A0} 91.10}} & \multicolumn{1}{r|}{{\color[HTML]{7030A0} 94.69}} & \multicolumn{1}{r|}{{\color[HTML]{203764} 99.24}} & \multicolumn{1}{r|}{{\color[HTML]{203764} 99.71}} & \multicolumn{1}{r|}{{\color[HTML]{203764} 91.62}} & {\color[HTML]{203764} 95.49} \\ \midrule
\multicolumn{1}{|r|}{ConLM}                      & \multicolumn{1}{r|}{{\color[HTML]{7030A0} 99.28}} & \multicolumn{1}{r|}{{\color[HTML]{7030A0} 98.54}} & \multicolumn{1}{r|}{{\color[HTML]{7030A0} 93.23}} & \multicolumn{1}{r|}{{\color[HTML]{7030A0} 96.00}} & \multicolumn{1}{r|}{{\color[HTML]{203764} 99.38}} & \multicolumn{1}{r|}{{\color[HTML]{203764} 99.67}} & \multicolumn{1}{r|}{{\color[HTML]{203764} 93.23}} & {\color[HTML]{203764} 96.34} \\ \bottomrule
\end{tabular}
\end{table}

\begin{table}[h]
\centering
\caption{Experimental results for the two UVA baselines (ConLM and LexLM) using the 2021AA-ACTIVE datasets with different levels of lexical similarity\label{2021AA-results}}
\begin{tabular}{@{}|rrrrrrrrr|@{}}
\toprule
\multicolumn{9}{|c|}{\textbf{2021AA-ACTIVE}}                                                                                \\ \midrule
\multicolumn{1}{|c|}{}                           & \multicolumn{4}{c|}{{\color[HTML]{548235} \textbf{GEN\_ALL}}}                                                                                & \multicolumn{4}{c|}{{\color[HTML]{305496} \textbf{GEN\_TOPN\_SIM}}}                                                                                \\ \cmidrule(l){2-9}
\multicolumn{1}{|c|}{\multirow{-2}{*}{Baseline}} & \multicolumn{1}{c|}{accuracy}                    & \multicolumn{1}{c|}{precision}                    & \multicolumn{1}{c|}{recall}                       & \multicolumn{1}{c|}{F1}                           & \multicolumn{1}{c|}{accuracy}                    & \multicolumn{1}{c|}{precision}                    & \multicolumn{1}{c|}{recall}                       & \multicolumn{1}{c|}{F1}      \\ \midrule
\multicolumn{1}{|r|}{LexLM}                      & \multicolumn{1}{r|}{{\color[HTML]{548235} 99.16}} & \multicolumn{1}{r|}{{\color[HTML]{548235} 87.14}} & \multicolumn{1}{r|}{{\color[HTML]{548235} 87.74}} & \multicolumn{1}{r|}{{\color[HTML]{548235} 87.44}} & \multicolumn{1}{r|}{{\color[HTML]{2F75B5} 97.74}} & \multicolumn{1}{r|}{{\color[HTML]{2F75B5} 88.22}} & \multicolumn{1}{r|}{{\color[HTML]{2F75B5} 87.74}} & {\color[HTML]{2F75B5} 87.98} \\ \midrule
\multicolumn{1}{|r|}{ConLM}                      & \multicolumn{1}{r|}{{\color[HTML]{548235} 99.51}} & \multicolumn{1}{r|}{{\color[HTML]{548235} 91.43}} & \multicolumn{1}{r|}{{\color[HTML]{548235} 93.96}} & \multicolumn{1}{r|}{{\color[HTML]{548235} 92.98}} & \multicolumn{1}{r|}{{\color[HTML]{2F75B5} 98.80}} & \multicolumn{1}{r|}{{\color[HTML]{2F75B5} 93.41}} & \multicolumn{1}{r|}{{\color[HTML]{2F75B5} 93.96}} & {\color[HTML]{2F75B5} 93.69} \\ \midrule
\multicolumn{1}{|c|}{}                           & \multicolumn{4}{c|}{{\color[HTML]{7030A0} \textbf{GEN\_RAN\_SIM}}}                                                                                & \multicolumn{4}{c|}{\textbf{GEN\_RAN\_NOSIM}}                                                                                \\ \cmidrule(l){2-9}
\multicolumn{1}{|c|}{\multirow{-2}{*}{Baseline}} & \multicolumn{1}{c|}{accuracy}                    & \multicolumn{1}{c|}{precision}                    & \multicolumn{1}{c|}{recall}                       & \multicolumn{1}{c|}{F1}                           & \multicolumn{1}{c|}{accuracy}                    & \multicolumn{1}{c|}{precision}                    & \multicolumn{1}{c|}{recall}                       & \multicolumn{1}{c|}{F1}      \\ \midrule
\multicolumn{1}{|r|}{LexLM}                      & \multicolumn{1}{r|}{{\color[HTML]{7030A0} 98.73}} & \multicolumn{1}{r|}{{\color[HTML]{7030A0} 98.73}} & \multicolumn{1}{r|}{{\color[HTML]{7030A0} 87.74}} & \multicolumn{1}{r|}{{\color[HTML]{7030A0} 92.91}} & \multicolumn{1}{r|}{{\color[HTML]{203764} 98.87}} & \multicolumn{1}{r|}{{\color[HTML]{203764} 99.69}} & \multicolumn{1}{r|}{{\color[HTML]{203764} 87.74}} & {\color[HTML]{203764} 98.33} \\ \midrule
\multicolumn{1}{|r|}{ConLM}                      & \multicolumn{1}{r|}{{\color[HTML]{7030A0} 99.28}} & \multicolumn{1}{r|}{{\color[HTML]{7030A0} 98.34}} & \multicolumn{1}{r|}{{\color[HTML]{7030A0} 93.96}} & \multicolumn{1}{r|}{{\color[HTML]{7030A0} 96.10}} & \multicolumn{1}{r|}{{\color[HTML]{203764} 99.40}} & \multicolumn{1}{r|}{{\color[HTML]{203764} 99.32}} & \multicolumn{1}{r|}{{\color[HTML]{203764} 93.96}} & {\color[HTML]{203764} 96.57} \\ \bottomrule
\end{tabular}
\end{table
\begin{table}[h]
\centering
\caption{Experimental results for the two UVA baselines (ConLM and LexLM) using the 2021AB-ACTIVE datasets with different levels of lexical similarity\label{2021AB-results}}
\begin{tabular}{@{}|rrrrrrrrr|@{}}
\toprule
\multicolumn{9}{|c|}{\textbf{2021AB-ACTIVE}}                                                                                \\ \midrule
\multicolumn{1}{|c|}{}                           & \multicolumn{4}{c|}{{\color[HTML]{548235} \textbf{GEN\_ALL}}}                                                                                & \multicolumn{4}{c|}{{\color[HTML]{305496} \textbf{GEN\_TOPN\_SIM}}}                                                                                \\ \cmidrule(l){2-9}
\multicolumn{1}{|c|}{\multirow{-2}{*}{Baseline}} & \multicolumn{1}{c|}{accuracy}                    & \multicolumn{1}{c|}{precision}                    & \multicolumn{1}{c|}{recall}                       & \multicolumn{1}{c|}{F1}                           & \multicolumn{1}{c|}{accuracy}                    & \multicolumn{1}{c|}{precision}                    & \multicolumn{1}{c|}{recall}                       & \multicolumn{1}{c|}{F1}               \\ \midrule
\multicolumn{1}{|r|}{LexLM}                      & \multicolumn{1}{r|}{{\color[HTML]{548235} 99.37}} & \multicolumn{1}{r|}{{\color[HTML]{548235} 88.18}} & \multicolumn{1}{r|}{{\color[HTML]{548235} 93.25}} & \multicolumn{1}{r|}{{\color[HTML]{548235} 90.64}} & \multicolumn{1}{r|}{{\color[HTML]{2F75B5} 98.31}} & \multicolumn{1}{r|}{{\color[HTML]{2F75B5} 89.13}} & \multicolumn{1}{r|}{{\color[HTML]{2F75B5} 93.25}} & {\color[HTML]{2F75B5} 91.14}          \\ \midrule
\multicolumn{1}{|r|}{ConLM}                      & \multicolumn{1}{r|}{{\color[HTML]{548235} 99.51}} & \multicolumn{1}{r|}{{\color[HTML]{548235} 91.52}} & \multicolumn{1}{r|}{{\color[HTML]{548235} 93.82}} & \multicolumn{1}{r|}{{\color[HTML]{548235} 92.65}} & \multicolumn{1}{r|}{{\color[HTML]{2F75B5} 98.82}} & \multicolumn{1}{r|}{{\color[HTML]{2F75B5} 93.60}} & \multicolumn{1}{r|}{{\color[HTML]{2F75B5} 93.82}} & {\color[HTML]{2F75B5} \textbf{93.71}} \\ \midrule
\multicolumn{1}{|c|}{}                           & \multicolumn{4}{c|}{{\color[HTML]{7030A0} \textbf{GEN\_RAN\_SIM}}}                                                                                & \multicolumn{4}{c|}{{\color[HTML]{203764} \textbf{GEN\_RAN\_NOSIM}}}                                                                                \\ \cmidrule(l){2-9}
\multicolumn{1}{|c|}{\multirow{-2}{*}{Baseline}} & \multicolumn{1}{c|}{accuracy}                    & \multicolumn{1}{c|}{precision}                    & \multicolumn{1}{c|}{recall}                       & \multicolumn{1}{c|}{F1}                           & \multicolumn{1}{c|}{accuracy}                    & \multicolumn{1}{c|}{precision}                    & \multicolumn{1}{c|}{recall}                       & \multicolumn{1}{c|}{F1}               \\ \midrule
\multicolumn{1}{|r|}{LexLM}                      & \multicolumn{1}{r|}{{\color[HTML]{7030A0} 99.27}} & \multicolumn{1}{r|}{{\color[HTML]{7030A0} 98.89}} & \multicolumn{1}{r|}{{\color[HTML]{7030A0} 93.25}} & \multicolumn{1}{r|}{{\color[HTML]{7030A0} 95.98}} & \multicolumn{1}{r|}{99.38}                        & \multicolumn{1}{r|}{99.73}                        & \multicolumn{1}{r|}{93.25}                        & 96.38                                 \\ \midrule
\multicolumn{1}{|r|}{ConLM}                      & \multicolumn{1}{r|}{{\color[HTML]{7030A0} 99.26}} & \multicolumn{1}{r|}{{\color[HTML]{7030A0} 98.18}} & \multicolumn{1}{r|}{{\color[HTML]{7030A0} 93.82}} & \multicolumn{1}{r|}{{\color[HTML]{7030A0} 95.95}} & \multicolumn{1}{r|}{99.40}                        & \multicolumn{1}{r|}{99.38}                        & \multicolumn{1}{r|}{93.82}                        & 96.52                                 \\ \bottomrule
\end{tabular}
\end{table}

We used the dataset generator described in Section \ref{generator} to generate UVA datasets.
We installed three UMLS releases including 2020AA, 2021AA, and 2021AB. We note that the 2022AA release has not been available at the time of this writing. For each UMLS release, we only selected active subset of source vocabularies in English, and the terms not being suppressed.

Table \ref{statistics} shows the statistics of the training and generalization test sets with the number of positive, negative, and total pairs for each UMLS version. These datasets will be used for the experiments described in Section \ref{experiments}.
This generator task required approximately 10,000 CPU cores from 300 to 600 nodes with 90 to 120 GB of RAM per node in the \textit{norm} partition of the Biowulf cluster, but finished within 24 hours.

\subsection{Generating UVA baselines}
\label{generating_baselines}

Here our goal is to demonstrate the reusability and the reproducibility of the baseline resources across multiple UMLS releases. Therefore, we did not perform extensive set of experiments with all variants from each UVA baseline. Instead, we chose the best variant of each baseline from previous experiments \cite{nguyen2022conlm,nguyen2021biomedical} to run the experiments.

For training the LexLMs and ConLMs, we use the TRAIN\_ALL dataset from each UVA version. For testing the overall generalization of each model, we used the GEN\_ALL. For testing the models with different levels of lexical similarity, we used the three corresponding datasets GEN\_TOPN\_SIM, GEN\_RAN\_SIM, and GEN\_RAN\_NOSIM. In the ConLMs, we configured to use ComplEx for training the ConKG embeddings.
We used default hyper parameters for the training and testing of the ConLMs and the LexLMs.

We report the experimental results in Table \ref{2020AA-results}, \ref{2021AA-results}, and \ref{2021AB-results} for each UVA dataset version.
Overall, the performance results of the ConLM, LexLM, and RBA are consistent with the ConLM performing best across the experiments.

\section{Discussion and Future Work}
\label{discussion}
We have presented the UVA resources across multiple UMLS releases to be shared with broader research communities. Here we discuss the directions we could move forward with these resources.

\subsection{Scalability, Reusability, and Reproducibility}

\textbf{Scalability.} Scalability is among the primary challenges at every step in the pipeline of the UVA problem. Every task we have deployed to the NIH's Biowulf HPC would require extensive resources. Especially, the tasks that require pair-wise calculation such as lexical similarity would be computationally heavy. Therefore, optimizing the tasks to run with less resource-extensive requirements within a reasonable timeframe remains a challenge for the project.

Additionally the scalability in terms of data size also increases when we extend the UVA task for the new applications. For instance, inserting a new vocabulary into the UMLS Metathesaurus would involve pair-wise predictions from the old and new terms.

\textbf{Reusability.} Our resources can be reused in different ways. The dataset generator can be reused for generating new UVA datasets from customized installations of the UMLS. For example, one can choose to install the full set of source vocabularies instead of just selecting the active ones. The data generator can also be extended to generate datasets with new characteristics. For example, we could limit the pairs in the UVA task to be from the same semantic group. Our baselines can be extended for developing the new approaches and even benchmarking them. Indeed, our LexLM has been reused for developing the ConLM. The resulting embeddings from our models can also be used for evaluating other BioNLP tasks such as UMLS entity recognition.

\textbf{Reproducibility.} Our resources presented in this paper are completely reproducible across multiple UMLS releases. We made the source code, the datasets, and other materials available \footnote{https://w3id.org/uva} for download. We also provide the scripts with customized parameters for automating the entire experiments.

\subsection{UVA Resource Maintenance Plan}
\label{releases}

\textbf{License.}
UMLS license is a no-cost license, required to download any UMLS distribution.
Therefore, the UMLS license is also required for using the UVA resources.

\textbf{UVA Challenge.} We plan to organize an annual challenge inviting new ideas, novel approaches and solutions for the UVA task. We will adopt the UVA resources to generate new datasets and baselines to serve as input for the UVA challenge.

The UVA task differs from the Ontology Alignment Evaluation Initiative (OAEI) in terms of scalability and domain diversity. Particularly, the size of the UVA datasets is several orders of magnitude more than tens of thousands of pairs in the OAEI datasets). Furthermore, the UVA task covers all the biomedical domains in the UMLS while the OAEI only cover a few ontologies. Therefore, we believe that the UVA task will complement the OAEI and offer greater challenges for the community.

\section{Related Work}
\label{related}

Biomedical ontology alignment is a long-standing research effort driven by the Ontology Alignment Evaluation Initiative (OAEI) since 2005. With the growth of interest in integrating biomedical ontologies at scale \cite{nguyen2021biomedical}, studies have looked into using rule-based and statistical approaches \cite{faria2013agreementmakerlight,ngo2016overview,jimenez2011logmap}, as well as supervised learning approaches for ontologies matching \cite{doan2004ontology}  by assessing the similarity \cite{kolyvakis2018biomedical,wang2018ontology} and relatedness \cite{mao2020use} between words and sentences. Such tasks are also known as Semantic Text Similarity (STS) tasks.

Recent progress has been attributed to the use of a combination of knowledge-based similarity with deep learning techniques, such as word embeddings \cite{mihalcea2006corpus} for input feature representation, and Siamese Network \cite{mueller2016siamese,he2015multi,bento2020ontology,huang2020sentence} to learn the underlying semantics and structure. Unsupervised approaches, such as transformer-based mechanisms, have also been explored with a great degree of success \cite{devlin2018bert}. Nonetheless, these techniques are largely based on lexical features and require very large text corpora to learn from. In \cite{iyer2020veealign}, contextualized representations of concepts with ancestral, child, data property neighbor, and object property neighbor contexts are used to discover semantically equivalent concepts. In our approach, we exploit the graph structure of various types of contextual information through the use of KG embeddings.

There are several families of KG embedding techniques \cite{ji2020survey,sun2020benchmarking}. They aim to map entities and relations into low-dimensional vectors while capturing their structural and semantic meanings \cite{wang2017knowledge}, and have shown to benefit a variety of knowledge-driven tasks \cite{lin2018knowledge}. Since this is the first attempt to use the various types of contextual information in the UVA task at scale, we explored three of the popular classes of techniques because of their demonstrated success in various tasks: translational distance-based with TransE \cite{bordes2013translating}, and TransR \cite{lin2015learning}; semantic matching-based using RESCAL \cite{nickel2011three}, DistMult \cite{yang2014embedding}, HolE \cite{nickel2016holographic}, and ComplEx \cite{trouillon2016complex}; and neural network-based using ConvKB \cite{nguyen2017novel}. We evaluated their performance in the UVA task, but did not benchmark them against other forms of graph representation techniques \cite{ji2020survey,wu2020comprehensive}. Many embedding techniques listed in \cite{sun2020benchmarking} are not selected due to the specific characteristics of context in our datasets, e.g., having no attributes/literals (but 10 techniques in \cite{sun2020benchmarking} including AttrE \cite{trisedya2019entity} and KDCoE \cite{chen2018co} leveraging attributes), having English-only (but MTransE \cite{chen2016multilingual} leveraging multilingual), or very large (RDGCN \cite{wu2019relation} being not scalable).

\section{Conclusion}
\label{conclusion}

We have presented the set of UVA resources including the dataset generator, the datasets, and the baseline approaches to be shared with broader research communities. We have shown how the resources can be reused and reproduced across multiple UMLS releases, and gathered to collectively serve the UVA challenge.

\paragraph*{Resource Availability Statement:} UVA resources including the source code and the datasets are available from \url{https://w3id.org/uva}.

\paragraph*{Acknowledgement.} This research was supported in part by the Intramural Research Program of the National Library of Medicine (NLM), National Institutes of Health (NIH). This research was also supported in part by two appointments to the National Library of Medicine Research Participation Program. This program is administered by the Oak Ridge Institute for Science and Education through an inter-agency agreement between the U.S. Department of Energy and the National Library of Medicine. We are thankful to Miranda Jarnot for helping us to confirm the editorial rules used in the Metathesaurus. We are also thankful to our summer interns (Hong Yung Yip, Goonmeet Bajaj, Thilini Wijesiriwardene, and Vishesh Javangula) and collaborators (Amit Sheth and Srinivasan Parthasarathy) for contributing to the resources presented in this paper.

\bibliographystyle{abbrv}