\section*{APPENDIX}

\bibliographystyle{IEEEtran}

\section{Introduction}
Camera and Light Detection and Ranging (LiDAR) sensors are essential components of autonomous vehicles with complementary properties. A camera can provide high-resolution color information but is sensitive to illumination and lack of spatial data. A LiDAR can provide accurate spatial information at longer ranges and is robust to illumination changes, but its resolution is much lower than the camera and doesn't measure color. Fusing the measurements of these two sensors allows autonomous vehicles to have an improved understanding of the environment. In order to combine the data from different sensor modalities, it's essential to have an accurate transformation between the coordinate systems of the sensors. Therefore, calibration is a crucial first step for multi-modal sensor fusion.

In recent years, many LiDAR-camera calibration methods have been proposed. These methods can be categorized based on whether they are online \cite{Nagy2019}\cite{Wang2020}\cite{Zhu2020}\cite{Lv2020}\cite{Yuan2020}\cite{Chien2016} or offline \cite{Mishra2020},\cite{Mishra2020a} \cite{Owens2015} as well as whether they require a calibration target \cite{Mishra2020},\cite{Mishra2020a} or not \cite{Lv2020}\cite{Yuan2020}\cite{Chien2016}\cite{Zhao2021}. Target-based methods require carefully designed targets \cite{Zhang2004}, \cite{Pusztai2017}, \cite{Owens2015} while targetless methods use data from the natural environment to perform calibration. In this paper, we develop an effective targetless method.

Calibration algorithms can also be categorized into whether they are optimization-based methods or learning-based methods. Most traditional methods are optimization-based methods that try to minimize or maximize a metric \cite{Mishra2020},\cite{Mishra2020a},\cite{Kang2020},\cite{Taylor2015},\cite{Pusztai2017},]\cite{Zhao2021}. Meanwhile, learning-based methods construct and train neural network models to directly predict the calibration parameters \cite{Lv2020},\cite{Schneider2017},\cite{Zhao2021},\cite{Yuan2020}. Both target-based and targetless optimization-based methods must define features for calibration. Common features include edges\cite{Mishra2020},\cite{Mishra2020a},\cite{Kang2020},\cite{Iyer2018}, gradient\cite{Taylor2015}, and semantic information \cite{Nagy2019,Zhu2020,Wang2020}. Among these, semantic information is a higher-level feature that is available from human annotations or learning-based semantic segmentation model predictions \cite{Zhang2019}, \cite{Yu2018}.
Several methods have been proposed to utilize semantic information to calibrate LiDAR and camera. Nagy et al. \cite{Nagy2019} use a structure from motion (SfM) pipeline to generate points from image and register with LiDAR to create basis calibration and refine based on semantic information. Zhu et al. \cite{Zhu2020} use semantic masks of image and construct height map to encourage laser points to fall on the pixels labeled as obstacles. The methods mentioned above only use semantic information from images and also define other features to help calibration. Wang et al. \cite{Wang2020} proposed a new metric to calibrate LiDAR and camera by reducing the distance between the misaligned points and pixels based on semantic information from both image and point cloud.

In this paper, we also use semantic information from LiDAR and camera measurements. We propose to use mutual information as the metric to optimize. Mutual information is widely used in medical image registration \cite{Oliveira2014, Nan2020}. Pandey et al.\cite{Pandey2015} first proposed to apply it to LiDAR camera calibration. Their approach considers the sensor-measured surface intensities (reflectivity for LiDAR and grayscale intensity for camera) as two random variables and maximizes the mutual information between them. Inspired by \cite{Nan2020}, we use mutual information neural estimate (MINE) \cite{Belghazi2018} to estimate the mutual information. We also use matrix exponential to compute transformation matrix and use the kernel-based sampler to sample points from images based on projected LiDAR. These treatments allow us to propose a fully differentiable LiDAR camera calibration framework based on semantic information, and implement the algorithm using popular Deep learning libraries.
The major contributions of this dataset can be summarized as follows:
\begin{itemize}
  \item We propose an algorithm for automatic, targetless, extrinsic calibration of a LiDAR and camera system using semantic information.
  \item We show how to make the objective function fully differentiable and optimize it using the gradient-descent method.
  \item We introduce an initial calibration method using 2D MI-based image registration.
  \item We evaluate our method on a synthetic dataset from Carla simulator \cite{Dosovitskiy17}, as well as the real KITTI360 \cite{Xie2016CVPR} and RELLIS-3D \cite{jiang2020rellis3d} datasets.
\end{itemize}
\section{Methology}
\subsection{Algorithm}
\input{figures/initial}
\input{algorithm/3D_calibration}

The following formula describes the transformation relationship of a point $p^L_i$ from the LiDAR local coordinate system to the camera projection plane:
\input{equations/lidar2cam2}
This paper focuses on the extrinsic calibration between LiDAR and camera; therefore, we assume that the camera and LiDAR have been intrinsically calibrated. The extrinsic calibration parameters are given by $R$ and $t$ in Eq.\ref{eq:lidarcam}. In addition to point cloud coordinates and image, we also assume the semantic labels of point cloud $L^L$ and image $L^C$ are available.

We consider the semantic label value of each point cloud and its corresponding image pixel as two random variables $X$ and $Y$. The mutual information of the two variable have the maximum value when we have the correct calibration parameters. In order to perform calibration, we  need to perform the following three operations:
\input{equations/operations}
 Therefore, our full optimization objective function can be written as Eq.\ref{eq:opt1}
\input{equations/opt1}
\subsection{Optimization}
The cost function Eq.\ref{eq:opt1} is maximized at the correct value of the rigid-body transformation parameters between LiDAR and camera. Therefore, any optimization technique that iteratively converges to the global optimum can be used, although we prefer gradient based optimization methods for their fast convergence properties. Here, we present how to make the cost function fully differentiable, which allows us to optimize Eq.\ref{eq:opt1} with a gradient-based optimization method. In the remaining part of this section, we describe how to make transformation matrix computation, image sampling and mutual information estimation differentiable, with the full algorithm given in Algorithm.\ref{alg:3d}.
\subsubsection{Transformation Matrix Computation}
$P^{uv} = \text{Proj}(P^{L},R,t)$ involves rigid 3D transformation $T$ which can be represented as the matrix exponential ($ T = \exp{(H)}$). And the matrix $H$ can be parameterized as a sum of weighted basis matrix of the Lie algebra $se(3)$ ($ H = \sum_{i=0}^{5} {v_iB_i}$) as described in \cite{Wachinger2013}. Therefore, we can represent $T$ with 6 parameters (see Eq.\ref{eq:expT}) and apply standard partial derivative to compute the gradient while optimizing.
\input{equations/expT}

\subsubsection{Image Sampling}
Differentiable image sampling is widely used in deep learning for computer vision \cite{Jaderberg2015}. The sampling operation can be written as
\input{equations/ksample}
where $\Phi_{x}$ and $\Phi_{y}$ are the parameters of a generic sampling kernel $k()$ which defines the image interpolation (e.g. bilinear), $l_{h w}^{C}$ is the label value at location $(h,w)$ of the image semantic label. $\widetilde{l}_{i}^{C}$ is the corresponding semantic label of point $p_i$ after projecting on image plane.
\subsubsection{Mutual Information Estimation}
Mutual information is a fundamental quantity for measuring the relationship between random variables. Traditional approaches are non-parametric (e.g., binning, likelihood-ratio estimators based on support vector machines, non-parametric kernel-density estimators), which are not differentiable. Several mutual information neural estimators have been proposed \cite{Belghazi2018, Mukherjee2019, KumarMondal}. These estimators are consist of neural networks that are fully differentiable. In our implementation,  we use MINE\cite{Belghazi2018} to estimate mutual information. This method uses the Donsker-Varadhan (DV) duality to represent MI as
\input{equations/mime},
where $P(X,Y)$ is the joint density for random variables $X$ and $Y$ and $P(X)$ and $P(Z)$ are marginal densities for $X$ and $Y$. $F_{\theta}$ is a function parameterized by a neural network, and $\theta$ is the parameters of the neural network.

\subsection{Initial Calibration}\label{sec:initial}
Most targetless calibration approaches require good initialization because good initialization helps the optimization converge faster. Besides, good initialization can help gradient-based optimization methods to avoid some local minimum. By utilizing the semantic information, we convert the initial calibration method into a combination of 2D image registration and Perspective-n-Point
(PnP) problem\cite{Lepetit2009}. We project LiDAR points into a spherical 2D plane \cite{Jiang2020} and get a 2D semantic range image from the LiDAR. We consider the LiDAR to be a low-resolution 360-degree camera with a partially overlapping field of view with the ordinary camera as shown in Fig.\ref{fig:initial}. Therefore, we zoom the semantic label of the camera image into the same resolution as LiDAR. Then, we perform 2D MI-based image registration on the two semantic images. After registration, we can get the raw correspondence between points of LiDAR and pixels of the camera image. Then, we sample several pairs of points and pixels and solve the PnP problem to get an initial calibration between the LiDAR and camera. The full algorithm is described in Algorithm.\ref{alg:init}

\input{algorithm/initial_calibration}

\section{Experiment and Results}
This section describes the experiments to evaluate the accuracy and robustness of the proposed automatic calibration technique. We first evaluate our methods on a synthetic dataset. Then we tested our methods on the real-world KITTI360 \cite{Xie2016CVPR} and RELLIS-3D \cite{jiang2020rellis3d} datasets.
\subsection{Synthetic dataset}
To test the accuracy and robustness of our methods, we created a dataset including paired LiDAR and camera data with semantic labels using the Carla simulator \cite{Dosovitskiy17}. The simulator can generate 21 classes. During, experiments we only used 20 classes. The image has a size of $1280\times720$. The Lidar has 360 degrees and 64 channels. Each channel includes roughly 800 points per frame. Due to the limitation of simulation, the point cloud and image are completely overlapped with each other (see Fig.\ref{fig:noise_label} (a)). However, the synthetic data can still provide the most accurate transformation and semantic labels. Another disadvantage of synthetic data is that some real-world sensor and environmental characteristics are not perfectly modeled.

We first tested the performance and robustness of our methods and the effects of the number of data frames on overall performance. In each test, we used the ground truth labels and provided the initial transformation through the methods described in section \ref{sec:initial}. Then, we tested the procedure with 1,10,20,50, and 70 pairs of frames. As shown in Fig.\ref{fig:boxplot}, the variance of the error decreases as we increase the number of pairs we use. Because by introducing more data, the estimator is able to have better mutual information estimation of the overall semantic information. Besides, more data also increase the scene's diversity and reduce the local minimum of the optimization.
\input{figures/noise_label}
Secondly, we tested the effects of the error of labels. In each test, we added random noise to both the image label and point cloud label (see Fig.\ref{fig:noise_label}). The results are shown in Fig.\ref{fig:boxplot_noise}. As expected, the accuracy decreases, and the variance increases with the error of the labels.
\input{figures/boxplot_noise}
\input{tables/kitti360}
\input{tables/rellis3d}
\input{figures/kitti360_example}
\input{figures/rellis3d_example}
\subsection{Real-world datasets}
Following our experiments with simulated data, we tested on two real-world datasets, KITTI360 and RELLIS-3D, which both provide annotated synchronized LiDAR and image data. Both datasets also provide calibration parameters between LiDAR and camera but they are not accurate as we can see in Fig.\ref{fig:kitti360} (a) and Fig.\ref{fig:rellis3d} (a). Therefore, we apply our methods on the two datasets and qualitatively compared with SOIC\cite{Wang2020}. We also follow Pandey's \cite{Pandey2015} method to use the mutual information between the sensor-measured surface intensities to calibrate the sensors. But we use MINE\cite{Belghazi2018} to estimate the MI.The results is noted as \textbf{PMI} in Table\ref{tab:kitti360} and \ref{tab:rellis3d}.All methods were initialized by our proposed method.  As shown in Table. \ref{tab:kitti360}-\ref{tab:rellis3d}, our method provide the closest results with the provided parameters of each dataset. Meanwhile, Fig. \ref{fig:kitti360} and \ref{fig:rellis3d} shows that our calibration results better cross-sensor data fusion than the other two methods. More visual results can be found in the video\footnote{\url{https://youtu.be/nSNBxpCtMeo}}.

\section{Summary and Future Work}
This paper presented a fully differential LiDAR-Camera calibration method using semantic information from both sensor measurements. By utilizing semantic information, the method doesn't need specific targets and initial guess parameters.
Because the method is fully differentiable, it can be implemented using the popular deep learning framework, such as Pytorch\cite{Paszke2019} or Tensorflow\cite{Abadi2016}. Moreover, mutual semantic information was introduced used to register multi-modal data. The method has the potential to leverage deep features to calibrate LiDAR and camera. Another possible application of this method is to use this framework in deep fusion directly. By embedding this framework in the deep fusion framework, rough calibration between sensors might be enough.

\section*{APPENDIX}

\bibliographystyle{IEEEtran}

\section{Introduction}
Camera and Light Detection and Ranging (LiDAR) sensors are essential components of autonomous vehicles with complementary properties. A camera can provide high-resolution color information but is sensitive to illumination and lack of spatial data. A LiDAR can provide accurate spatial information at longer ranges and is robust to illumination changes, but its resolution is much lower than the camera and doesn't measure color. Fusing the measurements of these two sensors allows autonomous vehicles to have an improved understanding of the environment. In order to combine the data from different sensor modalities, it's essential to have an accurate transformation between the coordinate systems of the sensors. Therefore, calibration is a crucial first step for multi-modal sensor fusion.

In recent years, many LiDAR-camera calibration methods have been proposed. These methods can be categorized based on whether they are online \cite{Nagy2019}\cite{Wang2020}\cite{Zhu2020}\cite{Lv2020}\cite{Yuan2020}\cite{Chien2016} or offline \cite{Mishra2020},\cite{Mishra2020a} \cite{Owens2015} as well as whether they require a calibration target \cite{Mishra2020},\cite{Mishra2020a} or not \cite{Lv2020}\cite{Yuan2020}\cite{Chien2016}\cite{Zhao2021}. Target-based methods require carefully designed targets \cite{Zhang2004}, \cite{Pusztai2017}, \cite{Owens2015} while targetless methods use data from the natural environment to perform calibration. In this paper, we develop an effective targetless method.

Calibration algorithms can also be categorized into whether they are optimization-based methods or learning-based methods. Most traditional methods are optimization-based methods that try to minimize or maximize a metric \cite{Mishra2020},\cite{Mishra2020a},\cite{Kang2020},\cite{Taylor2015},\cite{Pusztai2017},]\cite{Zhao2021}. Meanwhile, learning-based methods construct and train neural network models to directly predict the calibration parameters \cite{Lv2020},\cite{Schneider2017},\cite{Zhao2021},\cite{Yuan2020}. Both target-based and targetless optimization-based methods must define features for calibration. Common features include edges\cite{Mishra2020},\cite{Mishra2020a},\cite{Kang2020},\cite{Iyer2018}, gradient\cite{Taylor2015}, and semantic information \cite{Nagy2019,Zhu2020,Wang2020}. Among these, semantic information is a higher-level feature that is available from human annotations or learning-based semantic segmentation model predictions \cite{Zhang2019}, \cite{Yu2018}.
Several methods have been proposed to utilize semantic information to calibrate LiDAR and camera. Nagy et al. \cite{Nagy2019} use a structure from motion (SfM) pipeline to generate points from image and register with LiDAR to create basis calibration and refine based on semantic information. Zhu et al. \cite{Zhu2020} use semantic masks of image and construct height map to encourage laser points to fall on the pixels labeled as obstacles. The methods mentioned above only use semantic information from images and also define other features to help calibration. Wang et al. \cite{Wang2020} proposed a new metric to calibrate LiDAR and camera by reducing the distance between the misaligned points and pixels based on semantic information from both image and point cloud.

In this paper, we also use semantic information from LiDAR and camera measurements. We propose to use mutual information as the metric to optimize. Mutual information is widely used in medical image registration \cite{Oliveira2014, Nan2020}. Pandey et al.\cite{Pandey2015} first proposed to apply it to LiDAR camera calibration. Their approach considers the sensor-measured surface intensities (reflectivity for LiDAR and grayscale intensity for camera) as two random variables and maximizes the mutual information between them. Inspired by \cite{Nan2020}, we use mutual information neural estimate (MINE) \cite{Belghazi2018} to estimate the mutual information. We also use matrix exponential to compute transformation matrix and use the kernel-based sampler to sample points from images based on projected LiDAR. These treatments allow us to propose a fully differentiable LiDAR camera calibration framework based on semantic information, and implement the algorithm using popular Deep learning libraries.
The major contributions of this dataset can be summarized as follows:
\begin{itemize}
  \item We propose an algorithm for automatic, targetless, extrinsic calibration of a LiDAR and camera system using semantic information.
  \item We show how to make the objective function fully differentiable and optimize it using the gradient-descent method.
  \item We introduce an initial calibration method using 2D MI-based image registration.
  \item We evaluate our method on a synthetic dataset from Carla simulator \cite{Dosovitskiy17}, as well as the real KITTI360 \cite{Xie2016CVPR} and RELLIS-3D \cite{jiang2020rellis3d} datasets.
\end{itemize}
\section{Methology}
\subsection{Algorithm}
\input{figures/initial}
\input{algorithm/3D_calibration}

The following formula describes the transformation relationship of a point $p^L_i$ from the LiDAR local coordinate system to the camera projection plane:
\input{equations/lidar2cam2}
This paper focuses on the extrinsic calibration between LiDAR and camera; therefore, we assume that the camera and LiDAR have been intrinsically calibrated. The extrinsic calibration parameters are given by $R$ and $t$ in Eq.\ref{eq:lidarcam}. In addition to point cloud coordinates and image, we also assume the semantic labels of point cloud $L^L$ and image $L^C$ are available.

We consider the semantic label value of each point cloud and its corresponding image pixel as two random variables $X$ and $Y$. The mutual information of the two variable have the maximum value when we have the correct calibration parameters. In order to perform calibration, we  need to perform the following three operations:
\input{equations/operations}
 Therefore, our full optimization objective function can be written as Eq.\ref{eq:opt1}
\input{equations/opt1}
\subsection{Optimization}
The cost function Eq.\ref{eq:opt1} is maximized at the correct value of the rigid-body transformation parameters between LiDAR and camera. Therefore, any optimization technique that iteratively converges to the global optimum can be used, although we prefer gradient based optimization methods for their fast convergence properties. Here, we present how to make the cost function fully differentiable, which allows us to optimize Eq.\ref{eq:opt1} with a gradient-based optimization method. In the remaining part of this section, we describe how to make transformation matrix computation, image sampling and mutual information estimation differentiable, with the full algorithm given in Algorithm.\ref{alg:3d}.
\subsubsection{Transformation Matrix Computation}
$P^{uv} = \text{Proj}(P^{L},R,t)$ involves rigid 3D transformation $T$ which can be represented as the matrix exponential ($ T = \exp{(H)}$). And the matrix $H$ can be parameterized as a sum of weighted basis matrix of the Lie algebra $se(3)$ ($ H = \sum_{i=0}^{5} {v_iB_i}$) as described in \cite{Wachinger2013}. Therefore, we can represent $T$ with 6 parameters (see Eq.\ref{eq:expT}) and apply standard partial derivative to compute the gradient while optimizing.
\input{equations/expT}

\subsubsection{Image Sampling}
Differentiable image sampling is widely used in deep learning for computer vision \cite{Jaderberg2015}. The sampling operation can be written as
\input{equations/ksample}
where $\Phi_{x}$ and $\Phi_{y}$ are the parameters of a generic sampling kernel $k()$ which defines the image interpolation (e.g. bilinear), $l_{h w}^{C}$ is the label value at location $(h,w)$ of the image semantic label. $\widetilde{l}_{i}^{C}$ is the corresponding semantic label of point $p_i$ after projecting on image plane.
\subsubsection{Mutual Information Estimation}
Mutual information is a fundamental quantity for measuring the relationship between random variables. Traditional approaches are non-parametric (e.g., binning, likelihood-ratio estimators based on support vector machines, non-parametric kernel-density estimators), which are not differentiable. Several mutual information neural estimators have been proposed \cite{Belghazi2018, Mukherjee2019, KumarMondal}. These estimators are consist of neural networks that are fully differentiable. In our implementation,  we use MINE\cite{Belghazi2018} to estimate mutual information. This method uses the Donsker-Varadhan (DV) duality to represent MI as
\input{equations/mime},
where $P(X,Y)$ is the joint density for random variables $X$ and $Y$ and $P(X)$ and $P(Z)$ are marginal densities for $X$ and $Y$. $F_{\theta}$ is a function parameterized by a neural network, and $\theta$ is the parameters of the neural network.

\subsection{Initial Calibration}\label{sec:initial}
Most targetless calibration approaches require good initialization because good initialization helps the optimization converge faster. Besides, good initialization can help gradient-based optimization methods to avoid some local minimum. By utilizing the semantic information, we convert the initial calibration method into a combination of 2D image registration and Perspective-n-Point
(PnP) problem\cite{Lepetit2009}. We project LiDAR points into a spherical 2D plane \cite{Jiang2020} and get a 2D semantic range image from the LiDAR. We consider the LiDAR to be a low-resolution 360-degree camera with a partially overlapping field of view with the ordinary camera as shown in Fig.\ref{fig:initial}. Therefore, we zoom the semantic label of the camera image into the same resolution as LiDAR. Then, we perform 2D MI-based image registration on the two semantic images. After registration, we can get the raw correspondence between points of LiDAR and pixels of the camera image. Then, we sample several pairs of points and pixels and solve the PnP problem to get an initial calibration between the LiDAR and camera. The full algorithm is described in Algorithm.\ref{alg:init}

\input{algorithm/initial_calibration}

\section{Experiment and Results}
This section describes the experiments to evaluate the accuracy and robustness of the proposed automatic calibration technique. We first evaluate our methods on a synthetic dataset. Then we tested our methods on the real-world KITTI360 \cite{Xie2016CVPR} and RELLIS-3D \cite{jiang2020rellis3d} datasets.
\subsection{Synthetic dataset}
To test the accuracy and robustness of our methods, we created a dataset including paired LiDAR and camera data with semantic labels using the Carla simulator \cite{Dosovitskiy17}. The simulator can generate 21 classes. During, experiments we only used 20 classes. The image has a size of $1280\times720$. The Lidar has 360 degrees and 64 channels. Each channel includes roughly 800 points per frame. Due to the limitation of simulation, the point cloud and image are completely overlapped with each other (see Fig.\ref{fig:noise_label} (a)). However, the synthetic data can still provide the most accurate transformation and semantic labels. Another disadvantage of synthetic data is that some real-world sensor and environmental characteristics are not perfectly modeled.

We first tested the performance and robustness of our methods and the effects of the number of data frames on overall performance. In each test, we used the ground truth labels and provided the initial transformation through the methods described in section \ref{sec:initial}. Then, we tested the procedure with 1,10,20,50, and 70 pairs of frames. As shown in Fig.\ref{fig:boxplot}, the variance of the error decreases as we increase the number of pairs we use. Because by introducing more data, the estimator is able to have better mutual information estimation of the overall semantic information. Besides, more data also increase the scene's diversity and reduce the local minimum of the optimization.
\input{figures/noise_label}
Secondly, we tested the effects of the error of labels. In each test, we added random noise to both the image label and point cloud label (see Fig.\ref{fig:noise_label}). The results are shown in Fig.\ref{fig:boxplot_noise}. As expected, the accuracy decreases, and the variance increases with the error of the labels.
\input{figures/boxplot_noise}
\input{tables/kitti360}
\input{tables/rellis3d}
\input{figures/kitti360_example}
\input{figures/rellis3d_example}
\subsection{Real-world datasets}
Following our experiments with simulated data, we tested on two real-world datasets, KITTI360 and RELLIS-3D, which both provide annotated synchronized LiDAR and image data. Both datasets also provide calibration parameters between LiDAR and camera but they are not accurate as we can see in Fig.\ref{fig:kitti360} (a) and Fig.\ref{fig:rellis3d} (a). Therefore, we apply our methods on the two datasets and qualitatively compared with SOIC\cite{Wang2020}. We also follow Pandey's \cite{Pandey2015} method to use the mutual information between the sensor-measured surface intensities to calibrate the sensors. But we use MINE\cite{Belghazi2018} to estimate the MI.The results is noted as \textbf{PMI} in Table\ref{tab:kitti360} and \ref{tab:rellis3d}.All methods were initialized by our proposed method.  As shown in Table. \ref{tab:kitti360}-\ref{tab:rellis3d}, our method provide the closest results with the provided parameters of each dataset. Meanwhile, Fig. \ref{fig:kitti360} and \ref{fig:rellis3d} shows that our calibration results better cross-sensor data fusion than the other two methods. More visual results can be found in the video\footnote{\url{https://youtu.be/nSNBxpCtMeo}}.

\section{Summary and Future Work}
This paper presented a fully differential LiDAR-Camera calibration method using semantic information from both sensor measurements. By utilizing semantic information, the method doesn't need specific targets and initial guess parameters.
Because the method is fully differentiable, it can be implemented using the popular deep learning framework, such as Pytorch\cite{Paszke2019} or Tensorflow\cite{Abadi2016}. Moreover, mutual semantic information was introduced used to register multi-modal data. The method has the potential to leverage deep features to calibrate LiDAR and camera. Another possible application of this method is to use this framework in deep fusion directly. By embedding this framework in the deep fusion framework, rough calibration between sensors might be enough.