\subsection{A Human Matching Model}\label{sec:back}

The schema matching task revolves around providing correspondences among concepts, describing the meaning of data, \emph{e.g.,} database attributes. We present next a human matching model that has a static as well as dynamic components, the former is based on a model, presented by Gal~\cite{GAL2011}.

\subsubsection{Static Matching Model}
\label{sec:staticModel}
Let $S, S^{\prime}$ be two data sources with elements $\{a_1, a_2, \dots , a_n\}$ and $\{b_1, b_2, \dots , b_m\}$, respectively.
A matching process matches $S$ and $S^{\prime}$ by aligning their elements.

\begin{figure}[h]
	\vspace{-.4in}
	\begin{center}
		\includegraphics[width=\columnwidth]{./figs/S1_S2}
	\end{center}
	\vskip -.7in
	\caption{Schema Matching example.}
	\label{ex1}
\end{figure}
\begin{example}
	\label{ex:schemata1}
		We illustrate the model using the task of schema matching. Figure~\ref{ex1} presents two simplified purchase order schemata~\cite{DO2002a}. {\sf $PO_1$} has four attributes (foreign keys are ignored for simplicity): purchase order's  number ({\sf poCode}), timestamp ({\sf poDay} and {\sf poTime}) and shipment city ({\sf city}). {\sf $PO_2$} has three attributes: order issuing date ({\sf orderDate}), order number ({\sf orderNumber}), and shipment city ({\sf city}). A matching process aligns the schemata attributes, where a match is given by double-arrow edges, {\em e.g.}, {\sf orderNumber} in {\sf $PO_2$} corresponds to {\sf poCode} in {\sf $PO_1$}.
\end{example}

A matcher's output is conceptualized as a \emph{matching matrix} $M(S,S^{\prime})$ (or simply $M$), having entry $M_{ij}$ (typically a real number in $[0,1]$) represent a degree of alignment between $a_i\in{S}$ and $b_j\in{S^{\prime}}$.

A {\em match}, denoted $\sigma$, between $S$ and $S^{\prime}$ is a subset of $M$'s entries, containing of all non-zero entries. In our context we assume the existence of a ground truth as a matrix $M^e$, which represents a reference match such that $M_{ij}=1$ whenever the pair $(a_i,b_j)$ is part of the reference match and $M_{ij}=0$ otherwise. Reference matches were created to test matcher performance, typically compiled and refined by domain experts over time.

Matching is a complex decision making process, which involves a series of interrelated tasks~\cite{humanMatching}. Humans base their decisions on several aspects of the data source, such as attribute names, data-types, \emph{etc.} \add{Algorithmic matchers typically compute similarity between elements, which can be transformed using additional information (such as domain constraints) to report confidence.}
For human matchers, we can directly query their confidence level regarding a correspondence. We note, however, that these confidence values may hide judgment biases~\cite{tversky1974judgment}. An illustration of a matching matrix, using the use-case of Example~\ref{ex:schemata1}, is given in Figure~\ref{fig:ex1}.

\begin{figure}[h]
	\centering
	\vspace{-0.3in}
	\includegraphics[width=1.0\linewidth]{./figs/ex1}
	\vspace{-1.5in}
	\caption{Matching Matrix Generation Example}
	\label{fig:ex1}
\end{figure}

\subsubsection{Dynamic Matching Model}\label{sec:model_dyn}

Human matchers perform sequential decisions regarding element pairs and may change their mind, revisit previously determined matching decisions and assign a different confidence level to the same correspondence at different times. Therefore, we model the dynamic component of human matching decision making using a \emph{decision history} $H$, as follows. A decision regarding an element pair $\left(a_i, b_j\right)$ ($a_i\in{S}$, $b_j\in{S^{\prime}}$), is associated with a confidence $c\in [0,1]$ and a timestamp $t\in{\rm I\!R}$. A history $H=\langle h_t\rangle_{t=1}^{T}$ is a sequence of triplets of the form $\langle \left(a_i, b_j\right), c, t \rangle$. Each element in $H$ records a matching decision confidence $c$ concerning a pair of elements $\left(a_i, b_j\right)$ at time $t$. Timestamps induce a total order over $H$'s elements.

In addition, we use matchers' mouse movement to create a \emph{movement map} $G$. Each mouse movement (regardless of matching decisions) is associated with the visited position on the screen $\left(x, y\right)$, its type $v\in$ \{move ($\emptyset$), left click ($l$), right click ($r$), and scrolling ($s$)\} and a timestamp $t\in{\rm I\!R}$. A map $G=\langle g_t\rangle_{t=1}^{T}$ is a sequence of triplets $\langle \left(x, y\right), v, t \rangle$ representing a movement of type $v$ to position $\left(x, y\right)$ at time $t$. Aggregating the map  positions for each type creates a \emph{movement heat map} $G_{t}$, which is a screen size matrix where higher values are assigned to positions (pixels) that are frequently visited. Figure~\ref{fig:temp} illustrates two human matcher heat maps.

Given decision histories and movement maps, $\cal{D}$$=\{(H,G)\}$ denotes the space of human matchers.

\addtocounter{example}{-1}
\begin{example}[continued] Table~\ref{tab:ex1} provides a history excerpt of a human matcher, following Example~\ref{ex:schemata1}. The human matcher first matched {\sf PO$_1$.city} and {\sf PO$_2$.city} with a confidence level of $1.0$, at timestamp $3$, represented as the triplet $\langle M_{34}, 1.0, 3.0\rangle$. A decision regarding  {\sf PO$_1$.poDay} and {\sf PO$_2$.orderDate} was taken at time $8$ (with a confidence of $0.9$) and later lowered at time $16$ to a confidence of $0.5$, following an encounter with {\sf PO$_1$.poTime} (row 3 of Table~\ref{tab:ex1}).
\end{example}

\begin{table}[h]
	\caption{Human Matcher Decisions Example}\label{tab:ex1}
	\begin{center}
		\scalebox{1.2}{\begin{tabular}{|l||c|c|c|}\hline
				& \textbf{Entry} & \textbf{Confidence} & \textbf{Time}\\\hline\hline
				1 & $M_{34}$ & $1.0$ & $3.0$  \\
				2 & $M_{11}$ & $0.9$ & $8.0$  \\
				3 & $M_{12}$ & $0.5$ & $15.0$ \\
				4 & $M_{11}$ & $0.5$ & $16.0$ \\
				5 & $M_{21}$ & $0.45$ & $34.0$ \\\hline
		\end{tabular}}
	\end{center}
\end{table}

A matching matrix (Section~\ref{sec:staticModel}) is created from a matching history by assigning the latest confidence to each matrix entry. Referring to decision elements as $h.e$ (element pair), $h.c$ (confidence), and $h.t$ (timestamp), we compute a matrix entry as follows:

\begin{footnotesize}
	\begin{equation}\label{eq:hutomat}
	M_{ij} =
	\begin{cases}
	h_t.c | h_t.t = \max\limits_{h_t\in H | H_{t.e} = \left(a_i, b_j\right)}(h_t.t)
	& $\text{if }$ \exists h_t\in H |  h_{t.e} = \left(a_i, b_j\right) \\
	0.0 & \text{otherwise}\\
	\end{cases}
	\end{equation}
\end{footnotesize}

\subsection{A Model of a Matching Expert}\label{sec:ident}

Human matchers vary in their abilities. Ackerman {\em et al.}~\cite{humanMatching} show that human matchers may be biased in their decision making, which may lead to poor matching. Therefore, we seek a model of a matching expert, one we can rely on to be effective when making matching decisions. We consider human matchers to be ``weak experts" (rather than typical crowd sourcing workers that are assumed to be only ``generally knowledgeable"), satisfying some prerequisites such as familiarity with database systems.

Given a pair of data sources, $\left(S, S^{\prime}\right)$, we model an expert using measures of her observed performance, as captured by a matching matrix $M$ and a reference match $M^e$.  We focus on two measure types, namely quantitative (high quality) and cognitive (reliability). Specifically, on the quantitative level, an expert should be \emph{precise} and \emph{thorough}, and on the cognitive level she should be \emph{correlated} and \emph{calibrated}. These characteristics were chosen as representative of a wide range of requirements of a desirable matching system (see~\cite{GAL2011,humanMatching}). The proposed model can be extended and tuned to fit various matching system desiderata. For each measure we describe how to compute a matcher's expertise level, which when accompanied by a threshold ($\delta$) can determine expertise. Thresholds can be tailored to different expertise needs compiling differing system requirements.

 \subsubsection{Quantitative Measures} We first describe the two quantitative measures, namely {\em precision} and {\em thoroughness}, for achieving high-quality matches.

\vspace{0.1in}\noindent {\bf Precision:} A matching task involves multiple
\add{decisions regarding correspondences between schema elements}. Given a limited human attention span, a human expert is not expected to address all subtasks. However, we expect a matching expert to succeed in the subtasks she chose to address. We use the precision measure (Eq.~\ref{eq:P}, left) and set a threshold $\delta_{P}$ to capture a \emph{precise expert} (Eq.~\ref{eq:P}, right).
 	\begin{equation}
 	P(H)=\frac{\mid\sigma\cap M^{e+}\mid}{\mid\sigma\mid},  \expert{P}(H)= \mathbb{I}(P(H) > \delta_{P})
 	\label{eq:P}
 	\end{equation}
 	\noindent Recall that $\sigma$ is a subset of $M$'s entries. $M^{e+}$ represents the set of non-zero entries of $M^e$ and  $\mathbb{I}(\cdot)$ denotes an indicator function. $P(H)$ measures the ratio of correct matching decisions out of all matching decisions. $\delta_{P}$ was set to $0.5$ in the experiments to define a precise expert to be a matcher that matches correctly more pairs than she matches incorrectly.

\vspace{0.1in} \noindent {\bf Thoroughness:} Dealing with a complex task, and given limited span of attention, a human expert has to rely on her intuitions. Thus, human matchers may set a self-imposed time limit~\cite{humanMatching} and aim at covering more subtasks while sacrificing precision. We use recall (Eq.~\ref{eq:R}, left) with a $\delta_{R}$ threshold to define a \emph{thorough expert} (Eq.~\ref{eq:R}, right).

 	\begin{equation}
 	R(H)=\frac{\mid\sigma\cap M^{e+}\mid}{\mid M^{e+}\mid},  \expert{R}(H)= \mathbb{I}(R(H) > \delta_{R})
 	\label{eq:R}
 	\end{equation}

 	\noindent where $M^{e+}$, $\sigma$, and $\mathbb{I}(\cdot)$ are defined as before. $R(H)$ measures the number of correct matching decisions from all correct correspondences. Setting $\delta_{R} = 0.5$ represents an ability to cover most of the element pairs space as the number of identified correct correspondences exceeds the misidentified.

\addtocounter{example}{-1}
\begin{example}[continued] Let $H_{exp}$ be the matcher producing Table~\ref{tab:ex1}. Projecting a match for Table~\ref{tab:ex1} we obtain $\{M_{34}, M_{11}, M_{12}, M_{21}\}$.\footnote{Recall the $M_{ij}$ represents a correspondence between the $i$'th element in $S$ and the $j$'th element in $S^{\prime}$. For example, including $M_{11}$ in the match means that {\sf PO$_1$.poDay} and {\sf PO$_2$.orderDate} correspond.} Let $M^{e+} = \{M_{11}, M_{12}, M_{23}, M_{34}\}$ be the reference match for the matching problem of Figure~\ref{fig:ex1}, then, $P(H_{exp}) = \frac{3}{4}$, $\expert{P}(H_{exp}) = 1$, $R(H_{exp}) = \frac{3}{4}$, and $\expert{R}(H_{exp}) = 1$, leading to the conclusion that $H_{exp}$ is both precise and thorough.
\end{example}

\begin{figure}[h]
	\centering
	\includegraphics[width=.5\linewidth]{./figs/26_P_R_over_time}
	\includegraphics[width=.45\linewidth]{./figs/26}
	\caption{Matcher C: Precise and incomplete (not Thorough)}
	\label{fig:tempC}
\end{figure}

Recalling the matcher archetypes described in Section~\ref{sec:motivation}, Figure~\ref{fig:tempC} illustrates a third archetype, Matcher C. Similar to Matcher A (Figure~\ref{fig:expert}), Matcher C maintains a precise performance throughout the decision making process and her average confidence level generally follows the average precision level. However, in contrast to Matcher A, Matcher C fails to improve significantly her recall over time, resulting in an insufficient performance (less than $0.2$). Matchers of type C may be extremely confusing for contemporary human-in-the-loop matching systems as they seem (and actually are) very precise. Yet, with a limited timespan, matchers of type C cover only a fraction of the correct match. The heatmap shows that Matcher C mainly focuses on the top part of the right schema. This may indicate that Matcher C failed to reach the nested elements of the schema in the given timespan.

\subsubsection{Cognitive Measures} \label{sec:resulotion_calibration}

 To measure expert reliability, we apply state-of-the-art metacognitive measures~\cite{humanMatching} to matcher's reported confidence, assessing \emph{correlation} and \emph{calibration}. As both measures are computed relative to the entire matcher population, in the experiments we set thresholds with respect to the train set matchers.

 \vspace{0.1in}\noindent {\bf Correlation:} A \emph{correlated expert} is a matcher who is more confident when correct than when incorrect. We use resolution (Eq.~\ref{eq:Res}, top) to assess a correlated expert using a threshold $\delta_{Res}$ (left part of Eq.~\ref{eq:Res}, bottom). In addition, a matcher is considered correlated only if the resolution is statistically significant (right part of Eq.~\ref{eq:Res}, bottom).
	\begin{align}
	Res(H) = & \gamma(\sigma, M^{e+}),\nonumber\\
	\expert{Res}(H) = & \mathbb{I}(Res(H) > \delta_{Res} \wedge p_{val} < .05)
	\label{eq:Res}\end{align}
	\noindent where $\gamma(\cdot, \cdot)$ is a Goodman and Kruskal correlation.

\vspace{0.1in}\noindent {\bf Calibration:} A \emph{calibrated expert} is a matcher that can gauge her confidence. We use the calibration measure (Eq.~\ref{eq:Cal}) from metacognition research~\cite{ackerman2016metacognition}, accounting for over/under-confidence. Noting that better calibration is lower, we set a threshold $\delta_{Cal}$ to define a calibrated expert.
	\begin{equation}
	Cal(H) = \overline{H.c} - P(H),	\expert{Cal}(H) = \mathbb{I}(|Cal(H)| < \delta_{Cal})
	\label{eq:Cal}
	\end{equation}
\noindent where $\overline{H.c}$ is the average confidence reported by the user and $P(H)$ is her precision (Eq.~\ref{eq:P}).

In the experiments, we set thresholds to correspond to percentiles of the train population, setting $\delta_{Res}$ as the $80^{th}$ percentile and $\delta_{Cal}$ as the $20^{th}$ percentile.

\addtocounter{example}{-1}

\begin{example}[continued] Recall the matcher that produced Table~\ref{tab:ex1}. Based on the table, the calculated resolution is $1.0$ with $p_{val} = 0.5$. Although a resolution value of $1.0$ satisfies any $\delta_{Res}$, since $p_{val}>.05$, she is not considered correlated. The matcher's calibration is $0.67-0.75 = -0.12$, which means that she is under confident and given that the $20^{th}$ percentile in our experiments is $0.205$, she is considered calibrated.
\end{example}

\begin{figure}[h]
		\centering
		\includegraphics[width=.5\linewidth]{./figs/84_P_R_over_time}
		\includegraphics[width=.45\linewidth]{./figs/84}
		\caption{Matcher D: Precise, thorough, uncorrelated, and disorganized (not calibrated)}
		\label{fig:tempD}
\end{figure}

\begin{figure}[h]
	\centering
	\begin{subfigure}{0.45\linewidth}
	\includegraphics[width=\linewidth]{./figs/92_Res_Cal_over_time}
	\subcaption{Matcher A}
	\label{fig:tempRelA}
	\end{subfigure}
	\begin{subfigure}{0.45\linewidth}
	\includegraphics[width=\linewidth]{./figs/84_Res_Cal_over_time}
	\subcaption{Matcher D}
	\label{fig:tempRelD}
	\end{subfigure}
	\caption{Accumulated Resolution and Calibration of Matcher A (Figure~\ref{fig:expert}) and Matcher D (Figure~\ref{fig:tempD})}
	\label{fig:tempRel}
\end{figure}

Figure~\ref{fig:tempD} illustrates a fourth archetype, Matcher D, which like Matcher A (Figure~\ref{fig:expert}) maintains a precise and thorough performance, preserving high Precision values and consistently increasing Recall. However, the two matchers are different. To illustrate, we use Figure~\ref{fig:tempRel} to lay out the accumulated resolution (green bars) and calibration (pink bars) of matchers A and D. The pink bars in Figure~\ref{fig:tempRelD} represent the difference between the black dots (confidence) and blue bars (precision) of Figure~\ref{fig:tempD}. Figure~\ref{fig:tempRel} illustrates the questionable reliability of Matcher D. While Matcher A (Figure~\ref{fig:tempRelA}) provides a cognitively consistent behavior with a resolution close to $1.0$ and calibration close to $0.0$, Matcher D (Figure~\ref{fig:tempRelD}) fails to self-monitor its decisions resulting in a fairly low resolution (uncorrelated) and under confidence with a moderate absolute calibration value (not calibrated).

As a final note, as suggested by Ipeirotis {\em et al.}~\cite{ipeirotis2010quality}, predictable biased confidence levels, rather than low quality, may be manipulated to achieve much higher quality. For example, having realized that Matcher D is under-confident, a correspondence assigned with a $0.4$ confidence may be adjusted to $0.6$ and reconsidered as a part of the final outcome.

\subsection{Problem Definition - Expert Identification}\label{sec:prob}

Having introduced a matching expert model, we are interested in identifying such experts. Formally, let $\cal{Y} = $\{$+1$,$-1$\}$^{|L|}$, with $L$ being the expert characteristics space (in our case $|L| = 4$, with precision, thoroughness, correlation and calibration). Specifically, a $+1$ value for property $l\in 1,...,|L|$ represents expert ability, and $-1$ represents expert inability. Our problem definition can be expressed as follows.
 \begin{problem}\label{def:characterizer}
	Let $D = (H,G)$ be a human matcher representation and $Y$ be a matcher expert characteristics. We seek a \emph{matching expert characterizer $f: \cal{D}\rightarrow\cal{Y}$}, which is a mapping that maps $D$ into $Y$.
\end{problem}

\subsection{Human Matching Dataset}
\label{sec:dataset} The dataset contains $7716$ match decisions of $140$ human matchers, all Science/Engineering undergraduates who studied
database management courses. The study was approved by the institutional review board and four pilot participants completed the task prior to the study to ensure its coherence and instruction legibility. Participants were briefed in matching prior to the task, after which they were trained on a pair of short schemata (9-12 attributes) from the \emph{Thalia} dataset\footnote{\url{www.cise.ufl.edu/research/dbintegrate/thalia/howto.html}} prior to performing the main tasks.

The human matchers that participated in the experiments were asked to self-report personal information before the experiment.
The gathered information includes gender, age, psychometrics exam\footnote{\url{https://en.wikipedia.org/wiki/Psychometric_Entrance_Test}\label{fn1}} score, English level (scale of 1-5), knowledge in the domain (scale of 1-5) and basic database management education (binary).
The human matchers that participated in the experiments reported on psychometrics exam scores that are higher than the general population. While the general population's mean score is 533, participants average is 678. In addition, 88\% of human matchers consider their English level to be at least 4 out of 5 and only 14\% claim their knowledge in the domain is above 1. To sum, the participating human matchers represent academically oriented audience with a proper English level, yet with lack of any significant knowledge in the domain of the task.

The main matching tasks were chosen from two domains. The first is a \emph{Purchase Order} (PO) dataset~\cite{DO2002a} with schemata of medium size, with 142 and 46 attributes, and with high information content (labels, data types, and instance examples). The second domain is taken from an ontology alignment~\cite{EUZENAT2007a} task introduced in the OAEI 2011 and 2016 competitions,\footnote{\url{http://oaei.ontologymatching.org/2011/benchmarks/}} containing ontologies with 121 and 109 elements with high information content. The two tasks offer different challenges, where ontology elements differ in their characteristics from schemata attributes. Element pairs vary in their difficulty level, introducing a mix of both easy and complex matches.

Match confidence was inserted by participants as a value in $[0,1]$ to construct a history. We record the matcher mouse usage (clicks, moves, scrolls), accompanied by a timestamp and screen coordinates using Ghost-Mouse.\footnote{\url{https://www.ghost-mouse.com/}} Some preprocessing of the data was required to ensure the correctness of the results. This included removing the first three correspondences per participant, assuming a warm-up period is needed before response times are comparable. Of the $148$ participants, $8$ were discarded due to technical faults, leaving $140$ valid participants. Finally, elapsed time outliers (over 2 standard deviations from the mean of each participant) were removed due to the sensitivity of our measures to outliers. These outliers may be the result of methodical pauses by the participant, unrelated to the specific target term or revisiting a target term after time.

The interface that was used in the experiments is an upgraded version of the Ontobuilder research prototype~\cite{MODICA2001}, which is open source.\footnote{\url{https://github.com/shraga89/Ontobuilder-Research-Environment}} An illustration of the user interface is given in a technical report\footnote{\url{https://github.com/shraga89/MED/blob/master/MExI.pdf}}. Schemata are presented as foldable trees of terms (attributes). When selecting an attribute from the target schema, the match table presents a list of candidate attributes synchronized with the candidate schema tree. Selecting a term reveals additional information about it in a properties box. Terms that have sub-terms are highlighted. When a matcher selects an attribute, time until reaching a decision is recorded.

\subsection{Experimental Setup}
\label{sec:setup}

Evaluation was performed on a GPU server that contains two Nvidia gtx 2080 Ti and a CentOS 6.4 operating system. For the classifiers we used scikit-learn\footnote{\url{https://scikit-learn.org/stable/}} implementation and the networks were implemented using Keras\footnote{\url{https://keras.io/}} with a tensorflow backend. Adam~\cite{kingma2014adam} ($\eta = 0.001, \beta_1 = 0.9, \beta_2 = 0.999$) was used for optimization and cross entropy was used as a loss function. The code repository is available online.\footnote{\url{https://github.com/shraga89/MED}}

\subsubsection{Methodology} \label{sec:meth}
For the sequential feature extraction ($\Phi_{Seq}(H)$), following an LSTM hidden layer of $64$ nodes, we perform a $0.5$ dropout and a $100$ nodes fully connected layer with a ReLU activation. We fine-tuned a pre-trained ResNet~\cite{he2016deep} to extract spatial features ($\Phi_{Spa}(G)$). Finally, we add the label coefficients of each trained network to the feature-set (see Section~\ref{sec:class} for details).\footnote{Networks implementation is available at~\url{https://github.com/shraga89/MED/blob/master/utils.py}}

We present the results of two experiments. The first aims to quantify the ability of \emph{MExI} to identify experts in a schema matching task. The second aims at emphasizing the generalization abilities of \emph{MExI} using a related problem of ontology alignment. The experiments were conducted as follows:

{\bf Expert Identification experiment:} $106$ human matchers performed a schema matching task over the PO task (see Section~\ref{sec:dataset}), for which, we randomly split the matchers into $5$ folds and repeat an experiment $5$ times. For each experiment we use $4$ folds for training ($84$ matchers) and the remainder fold ($22$ matchers) for testing. In tables~\ref{tab:res1PO} and \ref{tab:res2} we report on the average performance over the $5$ experiments.\vspace{0.1in}

{\bf Generalizability experiment:} We use the $106$ PO task human matchers as a training set and the $34$ OAEI task (see dataset description above) human matchers as a test set. \vspace{0.1in}

For each experiment, we trained a set of state-of-the-art classifiers\footnote{Details in~\url{https://github.com/shraga89/MED/blob/master/Classifiers.md}} (\emph{e.g.,} SVM and Random Forest) for classification and selected the top performing classifier to be used for testing.

We evaluated three variations of the model. The first uses the set of human matchers as is (\add{\emph{MExI\_$\emptyset$}}). As part of the training phase (see Section~\ref{sec:meth}), we use sub-matchers to ensure sufficient data for a deep network. The sub-matchers were generated as a subset of consecutive decisions made by matchers in the training set and were used only during training. Specifically, we trained two additional models, \emph{MExI\_50} involves sub-matchers with 50 decisions each and \emph{MExI\_70} contains sub-matchers of $30, 40, \dots, 70$ decisions (average decisions per expert is 55).

\label{sec:base}\subsubsection{Baselines}

\add{We compared \emph{MExI} to seven baselines, using various methodologies for selecting high-quality individuals for a task. The first two baselines are fairly simple,  \textbf{\emph{Rand}} randomly assigns labels and \textbf{\emph{Rand\_Freq}} assigns labels by frequencies in the training set. Then, we introduce three baselines based on common practice quality control in crowdsourcing~\cite{daniel2018quality}. \textbf{\emph{Conf}} uses the reported confidence to determine expertise~\cite{oyama2013accurate}, \textbf{\emph{Qual. Test}} uses the warmup phase as a qualification test to estimate crowd`s accuracy following Zhang {\em et al.}~\cite{zhang2018reducing} which is used to determine expertise and \textbf{\emph{Self-Assess}} applies a pre-selection rule following Gadiraju {\em et al.}~\cite{gadiraju2017using}, where matchers with $|Cal|<0.2$ and $P>0.6$ during the warmup phase are classified as experts. Finally, we examine two learning-based baselines, which classify experts using matching predictors (\textbf{\emph{LRSM}}~\cite{lrsmTech}) and behavioral features as suggested by Goyal {\em et al.}~\cite{goyal2018your} (\textbf{\emph{BEH}}) .}

\label{sub:measures}\subsubsection{Evaluation Measures}
In our experiments we use two types of evaluation measures. First, we measure matching performance (Section~\ref{sec:regWithEx}) using precision, recall, resolution, and calibration (see Section~\ref{sec:ident}).
Second, to assess expert identification quality (Section~\ref{sec:mecEx}), we quantify accuracy with respect to a single characteristic (binary classification, Eq.~\ref{eq:acc1}) and all characteristics (multi-label classification, Eq.~\ref{eq:accMulti}). Let $\hat{Y}(D)$ and $Y(D)$ be the predicted and real characterization of $D$ respectively. Recall that $|\hat{Y}(D)| = |Y(D)| = |L|$; thus, we denote the $c$'th class (\emph{e.g.,} precise) as $Y_{c}(D)$ and $\hat{Y}_{c}(D)$. Then, accuracy for a single characteristic and all characteristics are defined as follows.
\begin{equation}\label{eq:acc1}
A_{c} = \frac{1}{K} \sum_{k=1}^{K} (Y_c(D_k) = \hat{Y}_c(D_k))
\end{equation}
\begin{equation}\label{eq:accMulti}
A_{ML} = \frac{1}{K} \sum_{k=1}^{K} \frac{Y(D_k) \cap \hat{Y}(D_k)}{Y(D_k) \cup \hat{Y}(D_k)}
\end{equation}

\subsection{Human Matcher Characterization}\label{sec:humanPop}
We start with an analysis of the overall population of matchers. Figure~\ref{fig:means} presents the mean performance of matchers using each of the four expertise measures (recall Section~\ref{sec:ident}). As illustrated, matchers are generally better in precision than recall (average of $0.55$ compared to $0.33$, respectively), which suggests that human matchers are geared towards correctness rather than coverage. Cognitively, the average resolution is relatively low in absolute value. However, when focusing on matchers with positive resolution (those that are more confident when correct), the average value is significantly higher ($0.61$ compared to $0.37$) indicating that positively correlated matchers offer better end result. Similarly, the average (absolute) calibration is deficient, \emph{i.e.,} the calibration is fairly high ($0.33$). Yet, focusing on under confident matchers, \emph{i.e.,} those with negative calibration, we obtain a much better average {\bf absolute} calibration of $0.11$ indicating that under confident matchers are more likely to be calibrated.

\begin{figure}[h]
	\centering
	\includegraphics[width=.85\columnwidth]{./figs/expert_means}
	\caption{Average performance of matchers by measure. Resolution ($Res$) and Calibration ($Cal$) are given in absolute value.}
	\label{fig:means}
\end{figure}

The proportion of matching experts by expertise type is illustrated in Figure~\ref{fig:counts}. Overall, more than half of the matchers are precise and only $\sim$15\% of matchers are thorough. This indicates again that human matchers aim to provide correct answers and are concerned less with the amount of responses they provide. 33\% of the matchers are correlated and 42\% of matchers are calibrated. Yet, as discussed above, positively correlated and under confident matchers are superior and 57\% of the former are correlated and 80\% of the latter are calibrated. Interestingly, 84\% of the under confident matchers are precise and 40\% are thorough (compared to 53\% and 15\%, respectively, over all matchers). This demonstrates the impact of cognitive measures on quantitative performance.

\begin{figure}[h]
	\centering
	\includegraphics[width=.85\columnwidth]{./figs/expert_counts}
	\caption{Proportion of matching experts by type. For each type, the bar represents the proportion of matchers that are experts. The darkest shade represents matchers that are experts in all four types and above it those that are experts in additional two and one type. The light shade at the top part of the bar represents the proportion of experts of this one type only. Note that all thorough experts ($\expert{R}$), are also experts in at least one additional expertise type.}
	\label{fig:counts}
\end{figure}

As a final note, we observe a correlation between reported English level and Recall and between reported psychometrics exam score and Precision. These results can be justified as better English speakers read faster and can cover more element pairs (Recall) and people that are predicted to have a higher likelihood of academic success at institutions of higher education (higher psychometric score) can be expected to be accurate (Precision). It is noteworthy that these are the only significant correlations found with personal information, and no significant correlation was found with resolution and calibration. This, in turn, emphasizes the importance of understanding the behavior of humans when seeking matching expertise, even when personal information is readily available.

\subsection{Characterizing Matching Experts}\label{sec:mecEx}

\begin{table*}[t]
	\centering
	\caption{\emph{MExI}'s accuracy compared to the baselines, using Eq.~\ref{eq:acc1} for $A_{P}, A_R, A_{Res}$ and $A_{Cal}$ and Eq.~\ref{eq:accMulti} for $A_{ML}$}
	\label{tab:res1}
	\begin{subtable}{0.475\textwidth}
		\subcaption{\textbf{Schema Matching (PO)}}\label{tab:res1PO}
		\scalebox{1.2}{\begin{tabular}{|l|c|c|c|c|c|}

				\hline
				$\downarrow$\textbf{Method}  & $A_{P}$                                & $A_{R}$       & $A_{Res}$     & $A_{Cal}$         & $A_{ML}$           \\
				\hline
				\textit{Rand}                  & .44                                    & .88           & .64           & .60               & .26                \\
				\textit{Rand\_eq}              & .56                                    & .88           & .64           & .40               & .12                \\
				\textit{Conf}                  & .35                                    & .70           & .57           & .40               & .18                \\
				\add{\textit{Qual. Test}}      & \add{.58}									& \add{.69}			& \add{.59}			& \add{.61}				& \add{.26}				 \\
				\add{\textit{Self-Assess}}     & .51                                    & .80           & .66           & .64               & .28                \\
				\textit{LRSM}                  & .80                                    & \textbf{.93}  & .71           & .73               & .33                \\
				\textit{BEH}                   & .81                                    & .88           & .64           & .70               & .30                \\
				\hline
				\textbf{\add{\emph{MExI\_$\emptyset$}}}         & \multicolumn{1}{l|}{.88}               & .88           & .71           & .80               & .46                \\
				\textbf{\textit{MExI\_50}} & \multicolumn{1}{l|}{\textbf{.98}$^*$ } & \textbf{.93}  & \textbf{.81}  & \textbf{.87}$^*$  & \textbf{.68}$^*$   \\
				\textbf{\textit{MExI\_70}} & \multicolumn{1}{l|}{.93$^*$ }          & .92           & .79           & .82               & .66$^*$            \\
				\hline
		\end{tabular}}
	\end{subtable}
	\quad\hspace{.25in}
	\begin{subtable}{0.45\textwidth}
		\subcaption{\textbf{Ontology Alignment (OAEI)}}\label{tab:res1OAEI}
		\scalebox{1.2}{\begin{tabular}{|l|c|c|c|c|c|}

				\hline
				$\downarrow$ \textbf{Method}  & $A_{P}$       & $A_{R}$       & $A_{Res}$     & $A_{Cal}$     & $A_{ML}$           \\
				\hline
				\textit{Rand}                  & .39           & .24           & .27           & .45           & .04                \\
				\textit{Rand\_eq}              & .60           & .75           & .27           & .55           & .18                \\
				\textit{Conf}                  & .58           & .73           & .39           & .58           & .17                \\
				\add{\textit{Qual. Test}}      & \add{.62}           & \add{.73}           & \add{.42}           & \add{.52}           & \add{.17}                \\
				\add{\textit{Self-Assess}}     & .61           & .76           & .43           & .55           & .17                \\
				\textit{LRSM}                  & .61           & .76           & .65           & .52           & .23                \\
				\textit{BEH}                   & .55           & .70           & .70           & .55           & .24                \\
				\hline
				\textbf{\add{\emph{MExI\_$\emptyset$}}}         & .61           & .74           & .70           & .57           & .31                \\
				\textbf{\textit{MExI\_50}} & \textbf{.70}  & \textbf{.83}  & \textbf{.74}  & \textbf{.61}  & \textbf{.43}$^*$   \\
				\textbf{\textit{MExI\_70}} & .61           & .76           & .73           & .61           & .35$^*$            \\
				\hline
		\end{tabular}}
	\end{subtable}
\end{table*}

Now, we turn our efforts to analyze the ability of \emph{MExI} to identify matching experts. Table~\ref{tab:res1} compares accuracy (eqs.~\ref{eq:acc1}-\ref{eq:accMulti}) results of \emph{MExI} to the baselines. An asterisk denotes statistical significant differences in performance using a two-sample bootstrap hypothesis test over the top performing baseline, LRSM ($p$-$value<.05$).

\add{Primarily,} using submatchers (\emph{MExI\_50}) boosts results, improving the PO task results (Table~\ref{tab:res1PO}) on $A_{P}$, $A_{R}$, $A_{Res}$, $A_{Cal}$, $A_{ML}$ by $11$\%, $5$\%, $14$\%, $9$\%, and $48$\% \add{over \emph{MExI}\_$\emptyset$}, respectively, indicating that using sub-matchers is a valuable approach. Yet, using it too aggressively, lowers accuracy, suggesting that reusing subsets with different sizes (as in \emph{MExI\_70}) is likely to overfit.

\add{\emph{MExI\_50} outperformed all baselines (statistically significant for $A_{P}, A_{Cal}$, and $A_{ML}$), suggesting that its ability to identify matching experts is superior to na\"ive approaches (\emph{Rand} and \emph{Rand\_Freq}), trusting human judgment (\emph{Conf}), using a qualification test (\emph{Qual. Test}), self-assessment based pre-selection (\emph{Self-Assess}), and recent literature (\emph{LRSM} and \emph{BEH}).}

Evaluating the generalizablity of \emph{MExI} (OAEI task, Table~\ref{tab:res1OAEI}), we observe a relatively smaller improvement over the baselines. Nevertheless, both \emph{MExI\_50} and \emph{MExI\_70} achieved a statistically significant improvement over the top performing baseline. Thus, as a proof-of-concept, we show that even when applying a trained \emph{MExI} over a new domain, it can still achieve high quality results improving the state-of-the-art.

\subsection{Feature Importance via Ablation Study}\label{sec:feaureSel}

Using \emph{MExI\_50} results over the PO dataset, we performed an ablation study to examine feature-sets influence. Table~\ref{tab:res2} reports on accuracy, comparing \emph{MExI\_50} to: 1) using each feature-set by itself (\emph{include}) and 2) removing a feature-set one at a time (\emph{exclude}). Boldface entries indicate the eminent feature-set with respect to an expert measure. For \emph{include} (\emph{exclude}), higher (lower) quality means higher importance.

\begin{table}[h]
	\centering
\caption{\emph{MExI} ablation study over the feature-sets. {\em include} refers to training using only one feature set while {\em exclude} refers to the exclusion of a single feature set at a time.}
\scalebox{1.1}{\begin{tabular}{|l|l|c|c|c|c|c|}
	\hline
	\multicolumn{2}{|l|}{$\downarrow$ \textbf{Method}} & $A_{P}$       & $A_{R}$       & $A_{Res}$     & $A_{Cal}$     & $A_{ML}$      \\
	\cline{1-7}
\multicolumn{2}{|l|}{\textbf{\textit{MExI\_50}} }  & \multicolumn{1}{l|}{\textbf{.98}} & \textbf{.93}  & \textbf{.81}  & \textbf{.87}  & \textbf{.68}   \\\cline{1-7}
	\multirow{5}{*}{\rotcell{\textbf{\emph{include} }}} & $\Phi_{LRSM}$                & \textbf{.80}  & \textbf{.93}  & \textbf{.71}  & .73           & .33           \\
	& $\Phi_{Mou}$                 & .68           & .88           & .56           & .52           & .32           \\
	& $\Phi_{Beh}$                 & .69           & .86           & .50           & .57           & .31           \\
	& $\Phi_{Seq}$                 & .77           & .78           & .66           & \textbf{.74}  & \textbf{.37}  \\
	& $\Phi_{Spa}$                 & .53           & .78           & .53           & .53           & .28           \\\cline{1-7}
	\multirow{5}{*}{\rotcell{\textbf{\emph{exclude} }}} & $\Phi_{LRSM}$                & \textbf{.81}  & \textbf{.85}  & .72           & .68           & .54           \\
	& $\Phi_{Mou}$                 & .86           & .87           & \textbf{.55}  & .72           & .58           \\
	& $\Phi_{Beh}$                 & .86           & .92           & .66           & .75           & .62           \\
	& $\Phi_{Seq}$                 & .83           & .88           & .66           & \textbf{.60}  & \textbf{.53}  \\
	& $\Phi_{Spa}$                 & .83           & .91           & .56           & .68           & .61           \\
	\cline{1-7}
\end{tabular}}
\label{tab:res2}
\end{table}

In terms of quantitative measures, $\Phi_{LRSM}$ is most important. For cognitive measures, mouse movement (mainly $\Phi_{Mou}$ and low accuracy without $\Phi_{Spa}$) and sequential decision process ($\Phi_{Seq}$) were predominant. This suggests that mouse movement implies whether an expert discriminates between the correct and incorrect decisions (correlation). Sequential decision process is mainly important to detect over-confidence (calibration). Finally, examining multi-label accuracy, results suggest that using LSTM to capture the expert sequential decision process ($\Phi_{Seq}$) is worthwhile and results in favorable performance even as a standalone feature-set.

\begin{table*}[h]
	\caption{Top 2 informative features for each feature set.$ ^{\ref{fn}}$}
	\label{tab:featureImp}
	\centering
	\scalebox{.8}{\begin{tabular}{|l|cc|cc|cc|cc|}
		\hline
		\textbf{Characteristic} $\rightarrow$& \multicolumn{2}{c|}{$\expert{P}$}                                   & \multicolumn{2}{c|}{$\expert{R}$}                                   & \multicolumn{2}{c|}{$\expert{Res}$}                              & \multicolumn{2}{c|}{$\expert{Cal}$}                                \\
		$\downarrow$ \textbf{Feature Set}     & (1)                                  & (2)               & (1)                                    & (2)             & (1)                                  & (2)            & (1)                                    & (2)            \\ \hline
		$\Phi_{LRSM}$               & \multicolumn{1}{c|}{dom}             & pca2              & \multicolumn{1}{c|}{pca1}              & normsinf        & \multicolumn{1}{c|}{bpm}             & normsinf       & \multicolumn{1}{c|}{pca2}              & bbm            \\
		$\Phi_{Mou}$                & \multicolumn{1}{c|}{totalLength}     & totalTime         & \multicolumn{1}{c|}{totalLength}       & avgX            & \multicolumn{1}{c|}{totalTime}       & totalLength    & \multicolumn{1}{c|}{totalLength}       & totalTime      \\
		$\Phi_{Beh}$                & \multicolumn{1}{c|}{avgTime}         & countDistinctCorr & \multicolumn{1}{c|}{countDistinctCorr} & avgTime         & \multicolumn{1}{c|}{countMindChange} & maxTime        & \multicolumn{1}{c|}{avgConf}           & maxTime        \\
		$\Phi_{Seq}$                & \multicolumn{1}{c|}{consensus ($P$)} & consensus ($Cal$) & \multicolumn{1}{c|}{consensus ($Cal$)} & consensus ($R$) & \multicolumn{1}{c|}{consensus ($P$)} & time ($P$)     & \multicolumn{1}{c|}{consensus ($Res$)} & time ($Cal$)   \\
		$\Phi_{Spa}$                & \multicolumn{1}{c|}{SMouse ($Res$)}  & SMouse ($P$)      & \multicolumn{1}{c|}{SMouse ($Res$)}    & Move ($R$)      & \multicolumn{1}{c|}{SMouse ($Res$)}  & LMouse ($Cal$) & \multicolumn{1}{c|}{Move ($Cal$)}      & SMouse ($Res$) \\ \hline
	\end{tabular}}

\end{table*}

Next, we analyzed feature importance within each feature-set\footnote{recall that the full list of features is given in \url{https://github.com/shraga89/MED/blob/master/Featuresets.md}\label{fn}} using SHAP~\cite{shap}. The two most important features in a feature set are given in Table~\ref{tab:featureImp} with the following main insights for each group\footnote{Top 5 most important features for each feature set is given in a technical report: \url{https://github.com/shraga89/MED/blob/master/MExI.pdf}}. For $\Phi_{LRSM}$, Dominance and PCA features were in the lead, supporting the feature analysis reported by Gal {\em et al.}~\cite{lrsmTech}, emphasizing uncertainty and diversity for expert identification. For aggregated features ($\Phi_{Mou}$ and $\Phi_{Beh}$), time and confidence are important along with the average screen position and the number of decisions and changed decisions, indicating that the main location (similar to ``on focus''~\cite{rzeszotarski2011instrumenting}) is important in determining expertise. In terms of sequential learning ($\Phi_{Seq}$), the consensus features (which were computed on the training set) were dominant across measures and the time and confidence features play a notable role as well. Finally, the scrolling features (SMouse), which may indicate uncertain behavior, were the most dominant for spatial learning ($\Phi_{Spa}$).

\subsection{Utilizing Matching Experts}\label{sec:regWithEx}

Finally, we analyze the impact the identification of matching experts has on matching quality.

\add{We begin by analyzing the average matching performance (in terms of $P$, $R$, $Res$, and $Cal$, see Section~\ref{sec:ident}) of the human matchers. We compare the performance of experts identified by \emph{MExI} (\emph{i.e.,} those that were identified as precise, thorough, correlated, and calibrated) to the full population of human matchers (\emph{no\_filter}) and the crowdsourcing quality assessment baselines (\emph{Conf}, \emph{Qual. Test} and \emph{Self-Assess}, see Section~\ref{sec:base}.)}

\begin{figure}[t]
	\centering
	\includegraphics[width=.8\linewidth]{./figs/quality_new.eps}
	\caption{\add{Performance (with error bars representing variance) of those identified experts by \emph{MExI} compared to the full population of matchers (\emph{no\_filter}) and crowdsourcing quality assessment baselines (\emph{Conf}, \emph{Qual. Test} and \emph{Self-Assess}), recalling that lower calibration is better.}}
	\label{fig:q}
	\vspace{-.15in}
\end{figure}

\add{Figure~\ref{fig:q} shows the quality of the identified experts obtained by the different methodologies. \emph{MExI}'s experts clearly outperform all baselines in terms of matching performance. Compared to \emph{no\_filter}, \emph{MExI} improved average precision by $42$\% (from $.55$ to $.78$), average recall by $90$\% (from $.29$ to $.55$), average correlation by $78$\% (from $.41$ to $.73$) and average calibration by $218$\% (from $.35$ to $.11$, recalling that lower calibration is better).} In a technical report\footnote{\url{https://github.com/shraga89/MED/blob/master/MExI.pdf}}, we show that even when using an expert geared towards a different measure, \emph{MExI} generates superior results with respect to using all human matchers. For example, thorough experts improve average precision by $53$\% (from $.55$ to $.84$).

\add{Human-in-the-loop systems can benefit from early identification, discharging non-experts sooner and preserving financial resources for future use. Therefore, we next analyze whether \emph{MExI} can assist in improving the final matching result using early detection of matchers as experts. We utilize \emph{MExI} to identify experts midway their decision process, and only use the identified experts.
We note that applying \emph{MExI} for early identification does not require labels for those decisions.}

\add{Figure~\ref{fig:qb} compares \emph{MExI}'s experts, determined using the first half of the median number of decisions per matcher ($30$), to the ones identified by crowdsourcing quality assessment methods and the full population of matchers. As illustrated, although the experts identified early achieve slightly inferior results than the ones identified at the end of the matching process (Figure~\ref{fig:q}), they still improve over all of the baselines.}

\begin{comment}
\subsubsection{Utilizing Matching Experts Types}\label{sec:regWithEx_tech}

Next, we compare average individual matching performance (in terms of $P$, $R$, $Res$, and $Cal$, see Section~\ref{sec:ident}) of all matchers, both those identified as experts and those identified as non-experts by \emph{MExI} against those that were identified as precise ($\expert{P}$), thorough ($\expert{R}$), correlated ($\expert{Res}$), and calibrated ($\expert{Cal}$) based on their history.

\begin{figure}[h]
\centering
\includegraphics[width=.8\linewidth]{./figs/quality.eps}
\caption{Performance (with error bars representing standard deviation) of those Identified as experts by \emph{MExI} per group.}
\label{fig:q_tech}
\end{figure}

Figure~\ref{fig:q_tech} shows the quality of the identified experts. Average precision is improved by $42$\% (from $.55$ to $.78$), average recall by $90$\% (from $.29$ to $.55$), average correlation by $78$\% (from $.41$ to $.73$) and average calibration by $218$\% (from $.35$ to $.11$). Even when using an expert geared towards a different measure, \emph{MExI} generates superior results with respect to using all human matchers. For example, thorough experts improve average precision by $53$\% (from $.55$ to $.84$).
\end{comment}

\subsection{Human Matching Features}\label{sec:feat}

We propose feature encoding of a human matcher, $\Phi: \cal{D}\rightarrow {\rm I\!R}$$^d$ that maps a pair of decision history $H$ and movement map $G$ (see Section~\ref{sec:back}) into a $d$ dimensional feature vector.

In what follows we suggest feature sets that serve at enhancing the ability to predict each of the four decision measures, grounding them in the relevant research areas. When designing the features, we make extensive use of the matching matrix (final decisions), the decision history (decision paths), and movement map (movement patterns). We note that some features may be predictive of more than one decision measure. Figure~\ref{fig:features} summarizes the proposed feature sets and full details are given in our repository.\footnote{\url{https://github.com/shraga89/MED/blob/master/Featuresets.md}}

\vspace{0.1in}\noindent {\bf Precision Features:} \emph{Matching predictors} were suggested for match evaluation when a reference match is unavailable~\cite{Sagi2013}. A matching predictor is a function that quantifies the quality of a match (given as a matching matrix). For example, a {\sf dominants} predictor measures the proportion of dominant element pairs, \emph{i.e.,} having highest value in their respective row and column. Matching predictors study yield observations regarding their varying usability towards precision or recall. Recently, matching predictors were suggested as features in learning to rerank schema matches (LRSM)~\cite{lrsmTech}. We use the matching matrix (computed from the history $H$, see Section~\ref{sec:model_dyn}) to generate matching predictors features, denoted as $\Phi_{LRSM}(H)$. We rely on Sagi {\em et al.}~\cite{Sagi2013} when choosing precision-oriented predictors.

\vspace{0.1in}\noindent {\bf Thoroughness Features:} Similarly to precision features, we use matching predictors to encode thoroughness features, focusing on predictors that were shown in the literature to lean towards higher recall. Specifically, predictors that capture negative characteristics such as uncertainty, diversity, and variability were shown to correlate with recall (and negatively correlate with precision). For example, matrix norm predictors~\cite{lrsmTech} are used to quantify the amount of error in the matching matrix, which can be attributed to uncertainty.

\vspace{0.1in}\noindent {\bf Correlation Features:} \emph{Match consistency} quantifies the extent to which a human matcher produces consistent matches~\cite{humanMatching}, highlighting human biases in the matching process. We create correlation features based on two consistency dimensions, namely, temporal and consensuality, which were shown to be predictive in terms of confidence and quality. The temporal dimension measures matching time and consensuality assesses the agreement among matchers. The predictive power of consistency analysis concerning confidence makes it effective for correlation features.

\vspace{0.1in}\noindent {\bf Calibration Features:} Calibration aims at qualifying matchers as experts by observing the dynamics of their matching. Therefore, calibration features naturally relate to the decision history $H$ and movement map $M$. Calibration features are grouped into three feature groups, as follows.

\textbf{Aggregated features} are extracted from the matching decision history ($\Phi_{Beh}(H)$) and the movement map ($\Phi_{Mou}(G)$). $\Phi_{Beh}(H)$ contains aggregations over confidence, decision times, and the number of changed matching decisions. For $\Phi_{Mou}(G)$, we follow~\cite{rzeszotarski2011instrumenting,wu2016novices,goyal2018your} to extact features.

\textbf{Sequential features ($\Phi_{Seq}(H)$)} examine the sequential decision making of a matcher through her declared confidence levels, the time spent until reaching a decision, and the extent of agreement with other matchers. \add{Sequential processing of the matching process aims to capture development (decline) in the matchers behavior}.

\textbf{Spatial features ($\Phi_{Spa}(G)$)} capture human matcher movement using its movement map $G$.\footnote{$\Phi_{Seq}(H)$ and $\Phi_{Spa}(G)$ are described in more details in the context of the algorithm in the next section.}

\subsection{Learning a Matching Expert Characterizer}\label{sec:class}

Equipped with a feature encoding for human matchers, we aim to find a ``good'' matching expert characterizer (Definition~\ref{def:characterizer}). We cast the problem as a multi-class multi-label classification problem. Following Read {\em et al.}~\cite{read2011classifier}, we transform the multi-label problem into a set of binary problems, one for each label. Hence, we train $|L|$ binary classifiers (one for each expert ability) using $\Phi(\cdot)$, where classifier $L_i$ is responsible for predicting $i$'s expert ability. The expert characterizer, $f$, returns $|L|$ binary labels, corresponding to the $|L|$ binary classifiers. Finally, given a (new) human matcher $D = (H, G)$, we extract $\Phi(H, G)$ and use the trained $f$ to characterize her, possibly identifying a new (unseen) expert.

The learning process is illustrated in Figure~\ref{fig:features} and is applied as follows. First, we capture the aggregated features, which can be calculated offline. Then, we employ neural networks to process the matching history $H$ sequentially with a recurrent neural network and the movement map $G$ spatially with a convolutional neural network.

Recurrent neural networks, and specifically long short-term memory (LSTMs), serve as a natural choice when processing the matching history sequentially. LSTMs use a gating system to control the amount of information to preserve at each timestamp using a hidden state. $\Phi_{Seq}(H)$ encodes the sequential decision making of a human matcher using her confidence levels ($h_1.c,\dots,h_T.c$), the time spent on a decision ($h_2.t-h_1.t,\dots,h_T.t-h_{T-1}.t$), and the level of agreement on a decision, $\pi_1,\dots,\pi_T$, with $\pi_i$ calculated as the number of human matchers in the training set that selected $h_1.e$ as part of their final matching matrix.

Given a human movement map, we seek a spatial analysis using a convolutional neural network (CNN), which was originally used for image processing, and apply convolution and pooling layers to extract filters over an input. $\Phi_{Spa}(G)$ encodes spatial matcher patterns of behavior through analysis of the main areas visited on the screen. We train four networks based on the movement heat maps $G_{\emptyset}$ (move over), $G_{l}$ (left click), $G_{r}$ (right click), and $G_{s}$ (scrolling). Since our dataset size is limited, we fine-tuned a CNN model, which was pre-trained on an image classification task~\cite{he2016deep}, with our dataset.

Finally, the set of trained models are fused as additional features to \emph{MExI}. Specifically, in this work, we adapt a late fusion strategy~\cite{snoek2005early}. During training, we first train the aforementioned networks on the training set of matchers. Then, we add label coefficients predicted by the networks as additional features ($\Phi_{Seq}(H)$ and $\Phi_{Spa}(G)$) to $\Phi(D)$ to train \emph{MExI}. During testing, we extract $\Phi_{Seq}(H)$ and $\Phi_{Spa}(G)$ using the trained networks, which are then added to $\Phi_{Beh}(H), \Phi_{Mou}(G)$ and $\Phi_{LRSM}(H)$ to construct $\Phi(D)$ and apply the trained \emph{MExI} to predict the labels.

\subsection{Motivating Example}\label{sec:motivation}
This work aims to characterize the expertise of human matchers using their behavioral profile.

To motivate the research into seeking matching experts, consider Figure~\ref{fig:temp}, which depicts two archetypes of human matchers based on our experiments (see Section~\ref{sec:dataset}). For each matcher, we measure accumulated Precision, Recall,\footnote{Precision measures the proportion of correct decisions out of made decisions and Recall is the proportion of correct decision out of all correct possible decisions (see Section~\ref{sec:ident}).} and average confidence ordered sequentially according to the order in which decisions were taken. In addition, we provide a mouse movement heat map.\footnote{Figures of all heat maps are given in \url{https://github.com/shraga89/MED/tree/master/Heatmaps}\label{heatmap}}

Figure~\ref{fig:expert} illustrates the performance of an expert. From the very beginning, Matcher A demonstrates precise decision making with high Precision values. In addition, many decisions are geared towards increasing the coverage of the match, adding more and more correct correspondences and incrementally increasing Recall. The average confidence, in this case, closely follows the aggregated precision values indicating that the matcher is cognitively-aware of decision odds. The heatmap shows Matcher A focuses on three main parts of the screen, the two schemata descriptions at the top and the matching matrix at the bottom.

Matcher B (Figure~\ref{fig:nonExpert}) represents typical performance of a non-expert. Starting at a low Precision level, Matcher B continues to make wrong matches, reducing Precision without increasing Recall much. As a result, the final performance measures remain fairly low. Matcher B demonstrates a significant over confidence, a well-known established human tendency, {\em e.g.},~\cite{Dunning2004,humanMatching}. It is interesting to see that the heatmap reveals that Matcher B has consistently refrained from investigating the metadata of the schema on the top left part of the screen, which may explain some of the poor performance.

\subsection{Contributions}
In this work, we aim at characterizing human matching experts, those humans whose proposed correspondences can be trusted to provide valid matches. We do so by offering {\em MExI} (Matching Expert Identification), a novel framework that learns matchers' qualification as experts based on their behavioral profile. Such a profile is composed of state-of-the-art matching predictors, aggregated behavioral features adapted from recent crowd quality assessment literature, and a novel use of neural networks to capture the decision making and mouse movements of human matchers. With such a tool at hand, we enhance the ability of matching systems to fuse appropriate experts and recognize their strengths and weaknesses when incorporating their input. \add{We demonstrate our approach using the task of schema matching and further show in our experiments its usefullness on the related task of ontology alignment.} Specifically, the paper provides the following specific contributions.

\begin{compactitem}
	\item We suggest a $4$-dimensional expert characterization framework, grounded in matching and metacognition research, to identify matching experts (Section~\ref{sec:ident}).
	\item We formulate expert matching identification as a classification problem, utilizing a novel set of features that stem from monitoring human matchers (Section~\ref{sec:mec}).
	\item We provide an extensive empirical evaluation to demonstrate the benefit of our approach. In particular, we show that {\em MExI} can identify and characterize matching experts dealing with challenging matching problem and the related problem of ontology alignment. In addition, we show the benefit of {\em MExI} in boosting final matching outcomes when identifying experts (Section~\ref{sec:eval}).
\end{compactitem}

Building blocks of our model are given in Section~\ref{sec:back} and related work in schema matching and assessing human expertise is discussed in Section~\ref{sec:related}. We conclude in Section~\ref{sec:con}.

\section{Introduction}\label{sec:intro}
	\input{intro}
	\section{Model and Problem Definition}\label{sec:back_prob}
	\input{back}
	\section{Identifying Matching Experts}\label{sec:mec}
	\input{features}
	\section{Empirical Evaluation}\label{sec:eval}
	\input{experiments}
	\section{Related Work}\label{sec:related}
	\input{related}
	\section{Conclusion and Future Work}\label{sec:con}
	\input{conclude}
	\balance
	\bibliographystyle{abbrv}


\section{Introduction} \label{sec:intro}
\label{sec1}
\noindent Severe disturbances in the near Earth's  space environment are caused by solar activity. The eruptive events like solar flares and coronal
mass ejections (CMEs) are often associated with radio emissions which originate in the solar corona. Such radio emissions or solar radio bursts (SRBs) describe various physical processes that happens in the solar corona and interplanetary medium (IPM). Based on the drifting
rates and morphology as seen in the dynamic spectrograms, SRBs are primarily classified into five classes vis. Type - I to V \citep{Wild1950, Wild1963}.
In this article, we study fast drifting Type III bursts, slow drifting Type II bursts and broad-band Type IV bursts as they provide clue to predict space weather \citep{Kundu1965, McLean1985, Pick2004}.\\\\
Type III bursts are the most intense, frequently observed and fast drifting bursts in the solar corona and IPM.
Most of the time, they occur in association with X ray and / or $H_{\alpha}$ flares \citep{Cane1988,White2007}.
Type III bursts occur as isolated bursts that lasts in 1 - 3s, in groups that last in 10 minutes and as storms that lasts in few hours.
The flare accelerated electrons that are travelling along open magnetic field lines setup the plasma oscillations (also known as Langmuir
waves) during their passage in the corona and IPM and subsequent conversion of those oscillations into electromagnetic waves produces the Type III bursts \citep{Ginzburg1958,Zheleznyakov1970,Melrose1980, Mercier1975,Pick1986, Sasikumar2013, Dayal2019, Mah2020, Reid2014, Saint_Hilaire_2012}.
In this article, we have carried out a statistical study of Type III bursts.\\\\
Type II bursts were discovered by \citet{Pay1947} and they are generated in association with CMEs that are moving at super-Alfvenic speeds \citep{Nelson1985,Cliver1999,Nindos2008,Nindos2011,Vrsnak2008}. Type II bursts are the signatures of particle acceleration caused by shock waves in the solar corona and IPM \citep{Gop2019}. Type IIs are slow drifting ($\rm \approx 1~MHz~s^{-1}$) bursts and it is
widely accepted that they are originated due to the plasma emission mechanism \citep{McLean1985,Nelson1985}.
Type II bursts often show fundamental and harmonic components with a frequency ratio $\approx 1:2$, and some times these components show band-splitting \citep{Vrs2001, Vas2014, Har2014, Har2015, Kis2016}. Type IV bursts are broadband quasi-continuum emissions that are mostly
associated with the flares/CMEs \citep{Boi1957, Ste1982, Gary1985, Ger1986, Gop1987, Sas2014, Car2017}.
Type IV bursts are further divided into two sub-categories: (i) stationary type IV bursts that occur during impulsive phase of the flares and the radio source location remains at same height, and (ii) moving type IV bursts which generally associated with CMEs. Imaging observations of moving Type IV bursts confirm that location of a radio source moves outward in the solar corona with speed of $\rm \approx 200 - 1500~ km~ s^{-1}$ \citep{Sas2014, Vas2016, Liu2018, Vas2019}. Their origin can be due to either plasma emission or gyro
synchrotron emission mechanism \citep{Sas2014,Car2017}. Both Type II and Type IV bursts show a significant correlation with space weather
hazards \citep{White2007, Vor2020}. \\\\
Type II, Type III, and Type IV bursts provide clues to predict the space weather hazards because: (i) Type III bursts are triggered by
solar flares, (ii) Type II burst originate from shocks observed in the solar corona and IPM, and (iii) moving Type IV bursts are
associated with the core of the CMEs \citep{Sas2014}. It is worth mentioning that enhancement of X ray and EUV radiation due to the
flares changes the conditions of the ionosphere and increases total electron content (TEC). Note that TEC of the ionosphere is a crucial
parameter related to the frequency of radio waves which experience transmission or reflection from the ionosphere \citep{Car2020, Sel2015}.
The CMEs and co-rotating interaction regions (CIRs) are responsible for geomagnetic storms and increased activity in the ionosphere.
Note that they impact the satellite communication, telecommunication, Global Navigation Satellite Systems (GNSS).
Also, the solar energetic particles damage the satellites and the radiation is hazardous for the astronauts and crew of the flights. \\\\
We make a note that X ray and EUV flux and energetic particles from the solar flares influence the ionosphere within hours and impacts
the HF communication whereas for CMEs to reach Earth takes 1 - 5 days depending on their speed and direction of arrival.
The observations shown in this article suggests that the radio observations carried out using ground based e-CALLISTO network can be
used to predict the space weather hazards.\\\\
A Compound Astronomical Low frequency Low cost Instrument for Spectroscopy and Transportable Observatory
(CALLISTO; \url{http://www.e-callisto.org/}) is a radio spectrometer designed to monitor the radio transient emissions 24 hours a day
\citep{Benz2009,Zuc2012, Sas2018}.
It is designed to operate in the frequency range 10 MHz -- 870 MHz and this frequency range probe the solar corona in the heliocentric
distance range $\rm \approx 1 - 3~R_\odot$ \citep{Poh2007}.
We make a note that there are $> 152$ stations operating around the globe and form an e-CALLISTO network.
Presently about 52 of them regularly provide data to a server (1 frame in every 15 minutes) located at the University of Applied Sciences
(FHNW) in Brugg/Windisch, Switzerland.
Therefore, routine detection of Type II, Type III and Type IV bursts along with other space based observations
(like GOES X-ray flux, Extreme Ultraviolet, and white light coronagraph images) are useful for space weather forecasting
agencies \citep{Prieto2020}. \\\\
This article is organized as follows.
In Section 2, we describe the data used in this article. Section 3 describes a statistical study of Type III bursts
(both isolated and group of Type III burst).
This section also deals with the space weather implication of Type II, Type III and Type IV bursts.
The summary and conclusion of the article are presented in Section 4.
\section{Observation}
\noindent In this study, we used the dynamic spectrograms observed using e-CALLISTO network \citep{Benz2005,Benz2009}.
Different stations operate over different frequency bands depending on the radio frequency interference (RFIs) and instrumental
limitations.
To begin with, we have prepared a list of Type III bursts using an online catalog ({\url{ftp://ftp.swpc.noaa.gov/pub/warehouse/}})
during 2010 - 2017 and manually cross-checked the bursts with the help of quick looks of the dynamic spectrograms that are available at
e-CALLISTO website  ({\url{http://soleil.i4ds.ch/solarradio/callistoQuicklooks/}}).
To cover the observations of 24 hours, we used data from different CALLISTO spectrometers that are listed in the Table \ref{Table1}.
Different columns of Table \ref{Table1} indicate the station-id, hosted country, corresponding longitude and latitude of the station, and the range of operating frequency, respectively. \\\\
\begin{table}[!htb]
\centering
 \caption{Description of CALLISTO Spectrometers used in this work. Note that a burst can be observed by more than one spectrometer}
 \label{Table1}
\begin{tabular}{|c|l| l | c| c|c|}
\hline \bf{SN} & \bf{File ID} & \bf{Country} & \bf{Lat ($^\circ$)} & \bf{Long ($^\circ$)} & Frequency range (MHz) \\
\hline 1 & BLEN7M     & Switzerland   & 46.94  &  7.45  & 170--870\\
\hline 2 & BIR        & Ireland       &  53.09 & -7.90  & 10--400 \\
\hline 3 & GAURI      & India         &  16.61 &  77.51 & 45--410 \\
\hline 4 & GLASGOW    & UK            & 55.90  & -4.30  & 45--80.9 \\
\hline 5 & GREENLAND  &  Greenland    & 67.00  & -50.72 & 10--110 \\
\hline 6 & BLENSW     & Switzerland   & 47.34  &  8.11  & 10 --90 \\
\hline 7 & ROSWELL-NM & New Mexico    & 33.39  & -104.52& 15--80.9\\
\hline 8 & SSRT       & Siberia       &  6.43  &  2.88  & 45--450\\
\hline 9 & KRIM       & Ukraine       &  44.95 &  34.10 & 250--350\\
\hline 10 & OOTY       & India        & 11.41  &  76.69 & 45--450 \\
\hline 11 & HUMAIN     & Belgium      & 50.20  &  5.25  & 45--410 \\
\hline 12 & ALMATY     & Kazakhstan   &  43.22 &  76.85 & 45--400\\
\hline 13 & MRT        & Mauritius    & -20.16 &  57.74 & 45--450\\
\hline 14 & RWANDA     & Rwanda       &  -1.94 &  30.06 & 45--80.9 \\
\hline 15 & ESSEN      & Germany      &  51.39 &  6.97  & 20--80.9  \\
\hline 16 & OSRA       & Czech Rep.   &  49.90 & 14.78  & 150--870\\
\hline 17 & TRIEST     & Italy        &  45.64 & 13.77  & 220--445\\
\hline 18 & ALASKA     & Alaska       &  64.20 & -149.49&  210--450\\
\hline 19 & DENMARK    & Denmark      &  55.67 &  12.56 & 45--100 \\
\hline 20 & eC71       & Switzerland  & 47.24  & 8.92   & 48--90 \\
\hline 21 & KASI       & South Korea  & 36.35  & 127.38 & 45--450 \\
\hline 22 & PERTH      & Indonesia    & -31.65 & 115.86 & 45--870\\
\hline 23 & DARO       & Germany      &  51.77 & 6.62   & 30--90 \\
\hline 24 & Heiterswil-CH& Switzerland & 47.30 & 9.13   & 45--82\\
\hline 25 & INDONESIA  & Indonesia    & -6.84  & 107.93 & 45-80.9 \\
\hline 26 & ROZTOKY    & Slovakia     &  49.40 & 21.48  & 45--400 \\
\hline
\end{tabular}
\end{table}
In Figure \ref{Fig1a}, left and right panels show the dynamic spectrogram of group of Type III bursts (henceforth Type IIIg
bursts) observed using a CALLISTO spectrometer located at Kigali, Rwanda (observed at 45 - 80 MHz) and at Royal Observatory of Belgium
(ROB; observed at 45 - 437 MHz), respectively. We make a note that since the observations at Rwanda are carried out over a narrow
bandwidth when compared it with the ROB, the frequency resolution of former instrument is better than the latter and thus the left
panel shows an intense and finer features. However, ROB observes the weak features of the burst beyond 80 MHz.\\\\
\begin{figure}[!ht]
 \centering
  \includegraphics[scale=.43]{Figure1a.eps}
  \includegraphics[scale=.43]{Figure1b.eps}
\caption{The dynamic spectrogram shows group of Type III bursts observed on 14 January 2015.
The left and right panels show the spectrograms observed using a CALLISTO spectrometer located at Kigali, Rwanda (45 - 80 MHz) and Royal
Observatory of Belgium (45 - 437 MHz), respectively. This event is associated with a flare of class C1.9.}
\label{Fig1a}
\end{figure}
For all the bursts, we measured the start and stop time, and lower ($F_{L}$) and upper ($F_{U}$) frequencies manually.
Although, different radio stations are operated over different range of operating frequencies (see Table \ref{Table1}), we selected $F_{L}$
and $F_{U}$ to be the lowest and highest frequencies among the simultaneous observations carried out by different stations, respectively.
Since Type III bursts are triggered by solar flares and some of the flares are accompanied by CMEs, we have carried out a statistical
analysis of them. The details of solar flares and CMEs and the associated radio bursts are taken from Heliophysics Event Catalogue
({\url{http://hec.helio-vo.eu/hec/hec\_gui.php}}).
If a radio burst is present between onset and end times of the flare then we treat that to be flare associated.
Otherwise, the burst is treated as non-flare associated.
\section{Results and Discussions}
\subsection{Statistical study of Type III bursts}
Many authors have attempted to study a correlation between occurrence of the total number of type III bursts and Sunspot number (SSN)
\citep{Lobzin2011, Huang2018, Mah2020}. Using the L-band observations, \citet{Huang2018} have studied 2384 SRBs observed
during 1997 - 2016 and reported that occurrence of SRBs closely track the solar cycle.
In the article, we have identified 12971 Type III bursts (during 2010-2017) that are reported by Space Weather Prediction Center (SWPC)
of National Oceanic and Atmospheric Administration (NOAA) and explored their relationship with the solar cycle.
\begin{figure}[!ht]
\centering
\includegraphics[scale=.99]{Figure2.eps}
\caption{Occurrence of Type III bursts and the solar activity. Occurrence rate of Type III bursts is well correlated with two peaks of
the Sunspots observed during solar cycle 24.}
\label{Fig2}
\end{figure}
Figure \ref{Fig2} shows the monthly averaged number of Type III bursts (i.e., number of Type III bursts observed in a year / 12)
observed in different years. The green curve shows the revised version of the Sunspot number that are available at {\url{http://www.sidc.be/silso/datafiles}} \citep{Clette2016}.
From Figure \ref{Fig2}, it is evident that occurrence of Type III bursts highly correlates with the solar cycle.
It is noteworthy that number of Type III bursts follow the two peaks of Sunspots that occurred during maximum period of solar cycle 24
(SC24; i.e., in the years 2012 and 2014). Also, a fewer number of bursts that are observed during the solar minimum
(i.e. in the years 2010 and 2017) are due to the lower solar activity.\\\\

Among the 12971 Type III bursts, we have randomly selected 619 intense type III bursts by visual inspection (including 221 isolated Type III and 398 Type IIIg bursts) and carried out a statistical study. We found that out of 619 Type III bursts, $\approx$65\% bursts are associated with soft and/or $H_\alpha$ flares while $\approx$ 45\% of
them are accompanied by CMEs (see Table \ref{tab:summary}).
A detailed table of type III bursts (that are used for statistical study) can be accessed from {\url{http://www.e-callisto.org/GeneralDocuments/Supplementary_material.pdf}}.
The remaining non-flare associated bursts presumably originated due to the weak energy releases that are present in the solar
corona \citep{Ramesh2010,Ramesh2013,Saint_Hilaire_2012,Sasikumar2013,Mugundhan2017,James2017,James2018,Sharma2018, Mah2020}.
Earlier \citet{Dayal2019} reported that out of 238 Type III bursts that are observed during $\approx$ 02:30 UT -- 11:30 UT in 2014,
88 bursts are associated with the GOES X ray flares / $H_\alpha$ flares.
\citet{Mah2020} reported that out of 1531 type III bursts, 426 bursts are associated with GOES X ray / $H_\alpha$ flares.
The non-flare associated Type III bursts may be triggered by  $H_{\alpha}$ ejecta, X-ray bright points, soft X-ray transient brightening,
and soft X-ray or/and extreme-UV (EUV) jets etc \citep[more references therein]{Alissandrakis2015}.
\begin{table}[h!]
\caption{Summary of observations}.
\label{tab:summary}
\centering
\begin{tabular}{lc lc lc}
\hline\hline
      & Isolated & Group of & Isolated + Group of   \\
Parameter      & Type III bursts & Type III bursts & Type III bursts   \\

\hline
Studied Type III bursts & 221 & 398 & 619 \\
Flare associated        & 135 & 267 & 402\\
CME accompanied          & 90 & 192 & 288 \\
Flare associated (\%)   & 61 & 67 & 65\\
CME accompanied (\%)     & 40 & 48 & 45 \\
\hline
\end{tabular}
\end{table}\\
Further, we have divided the total bursts into two categories and studied their characteristics separately.
The first category comprises a total of 221 isolated type III bursts (with duration $<$ 60 seconds) and the second category comprises
of 398 groups of type III bursts (with duration $\approx$ 1 min -- 10 min).
Figure \ref{Fig3} shows a CALLISTO spectrogram obtained at Bleien, Switzerland of an intense Type IIIg burst at the early
phase of flare of class X6.9 which is followed by another Type IIIg burst occurred during its decay phase.
Note that Type IIIg burst observed at 08:05 UT is associated with an ascending phase of flare while the burst at 08:17 UT is
associated with decay phase of the flare. The GOES X ray light curves are shown in black and red colors in Figure \ref{Fig3}.
\begin{figure}[h!]
\centering
 \includegraphics[scale=.5]{Figure3.eps}
 \caption{Dynamic spectrogram show two groups of Type III bursts. The one at 08:01 UT occurred during the ascending phase of X6.9
 solar flare. Type IIIg that occurred at 08:17 UT is triggered by the flare during its decay phase.
 The red and black curves shows the GOES X ray flux.}
 \label{Fig3}
\end{figure}\\
For all the isolated Type III bursts we measured the start time ($t_i$), stop time ($t_f$), upper frequency ($F_U$) and lower frequency
($F_L$) of the bursts and measured their drift rates using,
\begin{equation}
 \frac{df}{dt}=\left|\frac{F_{U}-F_{L}}{t_f-t_i}\right|
 \label{ED}
\end{equation}
Figure \ref{Fig4} illustrates the distribution of drift rates for different lower and upper frequencies.
From the plot it is found that at lower frequencies drift rates are smaller compared with the higher frequencies.
One possible explanation is, when the flare accelerated electrons travel along the open magnetic field lines which diverges with the
radially increasing distance, duration of the radio bursts increases and thus lower drift rates are expected at lower frequency
(i.e., in outer layers of corona).
\citet{Reid2014} interpret that faster drift rates in the deep solar corona is due to rapid change of frequency of the background
plasma with the increasing distance.

In this study, we have found that drift rates of the isolated bursts vary in the range $\rm 0.70 - 99~ MHz~s^{-1}$.
Earlier \citet{Zhang2018} have studied 1389 simple isolated Type III bursts observed using the Nan\c{c}ay decameter array over the
frequecy range 10 MHz - 80 MHz.
The authors have reported that drift rates of the Type III bursts range from $\rm 2 ~MHz~s^{-1}$ to $\rm 16~MHz~s^{-1}$, with a median
value of $\rm 6.94~ MHz~s^{-1}$. In Figure \ref{Fig4}, both color and the size of the marker indicate different drift rates for a given lower and upper frequency.

\begin{figure}[ht!]
\centering
\includegraphics[scale=.99]{Figure4.eps}
\caption{Distribution of drift rates of isolated Type III bursts measured for a given lower and upper frequencies.
Note that larger size of the marker indicate higher drift rates.}
\label{Fig4}
\end{figure}
We have studied the duration (i.e. stop time minus start time) of 398 Type IIIg bursts.
The left panel of Figure \ref{Fig5} shows the way duration of Type IIIg bursts are varied.
It is also evident that duration of Type IIIg bursts are independent of lower and upper frequency cut-offs.
Different colors and sizes of the markers indicate duration of Type IIIg bursts for a given lower and upper frequencies.
The histogram in right panel shows that 388 Type IIIg bursts (i.e. 97.5 \%) lasted within 4 minutes of time.
\begin{figure}[h!]
 \centering
  \includegraphics[scale=.55]{Figure5a.eps}
  \includegraphics[scale=.55]{Figure5b.eps}
\caption{The left panel shows the time duration over which Type IIIg bursts are lasted. The color and size of the marker indicate
the time interval over which they lasted. Note that larger the size of the marker means longer duration. The right panel shows number of
Type III bursts vs the time interval over which they are observed.}
\label{Fig5}
\end{figure}
\subsection{Space weather implication}
\noindent In this article, we investigated two events that caused the radio blackouts in different parts of Earth.
Using those events we attempt to explain the way radio emissions can help to predict the space weather hazards.
\subsubsection{The 6th September 2017 event}
On 6 September 2017, an intense and large group of Type III burst is observed at Greenland station, and its emission coincided with the
early phase of large flare of class X9.3 that peaked at 12:02 UT as shown in Figure \ref{Fig6}.
The explosion of this flare was stronger and the most intense in SC24. The eruption is caused by a complex Active Region (AR) 12673
located at S09W34. The green and red curves on Figure \ref{Fig6} are the GOES X-ray light curves. \\\\
The group of Type III bursts started at 11:57 UT and disappeared at 12:12 UT in the frequency range of 18 MHz -- 100 MHz.
Type IIIg bursts are observed during both ascend and decay phase of the flare.
Following Type IIIg, two Type II bursts and a stationary Type IV burst are observed. The first Type II burst is observed from
12:02:37 to 12:09:31 UT and in the frequency range 18 MHz - 100 MHz. The measured drift rate of the burst is $\approx 0.2~  MHz~s^{-1}$.
The second Type II burst is observed between 12:13:02-12:21:57 UT and in the frequency range 18 MHz - 60 MHz.
The measured drift rate is $\approx 0.1~ MHz~s^{-1}$. Another interesting observation is both Type II bursts show the fundamental and
harmonic emissions. Also, a Type IV burst is observed at 12:14 UT. \\\\
Here, we would like to emphasize the following points: (i) Type IIIg bursts are triggered by flare accelerated electrons, (ii) both
type II bursts are triggered by CME shocks, and (iii) Type IV burst is associated with CME. In other words, the radio bursts that
are observed in the Figure \ref{Fig6} are basically signatures of flares and/or CMEs.
Therefore, such observations can be used to forecast the space weather even in the absence of white light coronagraph observations.
Also, since there are $> 152$ CALLISTO stations are operating around the globe to monitor the transient emissions from the solar corona,
e-CALLISTO network is a powerful tool to forecast the space weather hazards.
Note that the CMEs associated with the radio emission shown in Figure \ref{Fig6} is Earth-directed and a nearly symmetrical halo with an
estimated sky-plane velocity of 1571 km s$^{-1}$ as observed by SOHO coronagraph at 12:24 UT.
Also, the CME caused a significant compression to the day side Earth's magnetosphere and prompted a severe (G4) geomagnetic storm
($\rm K_{pmax}= 8.3$ and $\rm Dst_{min}= -124$ nT) that reached the Earth on 7-- 8 September 2017 (see Figure \ref{Fig6}).

\begin{figure}[h!]
\centering
\includegraphics[scale=.5]{Figure6a.eps}
\includegraphics[width=0.7\textwidth]{Figure6b.eps}
\caption{The top panel shows radio emissions that are observed on 6 September 2017. The red and green curves show a GOES X ray flux
during the radio observations.
It shows Type IIIg burst during rising and decay phase of the X ray flare and are followed by two Type II and a stationary Type IV burst.
Lower panel shows a radio blackout that occurred over Europe, Africa and the Atlantic Ocean on 6 September 2017 due to the enhancement of
X-ray flux and UV radiation at the Earth. Credited to SWPC.}
  \label{Fig6}
  \end{figure}
Furthermore, intense radio emissions shown in Figure \ref{Fig6} are associated with major impulsive increases with X-rays and EUV
emission.
\citet{Sato2019} have reported the impact of the 6 September 2017 on GNSS signals and inferred that the bursts are strong enough to give
rise sudden disturbances on space weather environment.
The EUV emission is caused a prompt ionization enhancement in the Earth's upper atmosphere. Also the energetic particles from the Sun
arrived at the Earth within few hours after the flare and resulted a large enhancement of high energy proton levels and caused a shortwave
radio blackout over Europe, Africa and the Atlantic Ocean as shown in lower panel of Figure \ref{Fig6}.
Moreover, a rapid and comprehensive ionization of the equatorial upper atmosphere has disrupted HF communications
while emergency managers were struggling to provide critical recovery services (e.g., \citealp{NCEI2017}).
Some of these issues were reported by the Hurricane Weather Net (HWN), and the French Civil Aviation Authority (DGAC) \citep{Redmon2018}.
\subsubsection{The 7th March 2010 event}
Type IIIg burst and a stationary type IV burst shown in Figure \ref{Fig7} is observed by Badary observatory (SSRT site), Siberia,
Russian Federation on 7 March 2012.
These radio emissions are triggered by GOES X5.4 flare and its onset and peak times are 00:02 UT and 00:24 UT, respectively.
This flare is associated with active region AR11429 located at N18E31. The second flare is triggered by GOES X1.3 class flare and its
onset and peak times are 01:05 UT and 01:14 UT, respectively. This flare also associated with an active regions AR11429 located at N15E26.
The green and red curves in Figure \ref{Fig7} show the X ray light curves observed by GOES satellite.\\\\
Type IIIg burst observed at 00:56 UT in the frequency range of 90 MHz -- 320 MHz is followed with an intense Type IV burst.
This Type IIIg is triggered by X5.4 class flare during its decay phase. Another Type IIIg burst occurred at 01:58 UT is triggered by X1.3
flare during its decay phase.
These flares are associated with two CMEs launched from AR11429 region. The estimated speeds of first and second CME are $\approx 2200$
and $\approx 1800~ km~s^{-1}$, respectively \citep{Patsourakos2016}.
This event caused radio blackouts in Asia and Australia, and Indian and Pacific Oceans on 07 March 2012 at 04:21 UT (Figure \ref{Fig7})
and is followed by the aurora. The main phase of the magnetic storm starts at $\approx$ 02:00 UT on 7 March 2012 and reached a maximum at
$\approx$ 05:15 UT, with Dst$=$ -74 nT and $\rm K_{pmax}= 6.0$ \citep{Tsurutani2014}.
The ionospheric disturbances caused by this storm is described by \citet{Krypiak2019}.
\begin{figure}[h!]
\centering
\includegraphics[scale=.5]{Figure7a.eps}
\includegraphics[width=0.7\textwidth]{Figure7b.eps}

\caption{The top panel shows radio emissions observed on 7 March 2012 by Badary station in Russia.
The red and green curves show a GOES X ray flux during the radio observations.
It shows Type IIIg bursts during rising and decay phase of the X ray flare and a stationary type IV burst.
Lower panel shows a radio blackout that occurred over Asia, Australia, and Indian and Pacific Oceans on 7 March 2012 due to the
enhancement of X-ray flux and UV radiation at the Earth. Credited to SWPC.}
  \label{Fig7}
  \end{figure}
\section{Summary and conclusion}
\noindent In this study, we report a statistical analysis of Type III radio bursts and their correlation with the Sunspot number.
During solar cycle 24, we have identified 12971 Type III bursts (during 2010-2017) by making use of the reports of Space Weather
Prediction Center (SWPC) of National Oceanic and Atmospheric Administration (NOAA).
We found that the occurrence of Type III bursts well correlates with the Sunspot number.
Further, we have randomly selected 619 Type III bursts comprising 221 isolated and 398 group of bursts and carried out a statistical study of
Type III bursts using the dynamic spectrograms observed by e-CALLISTO network. We found that $65 \%$ of them are flare associated and
$45\%$ of the bursts are accompanied by a CME.
The remaining non-flare associated Type III bursts are believed to originate from weak energy release events in the solar atmosphere.
We have also found that drift rates of isolated bursts vary in the range $\rm 0.70 - 99~ MHz~s^{-1}$.
This study confirms that drift rates at high frequency are larger than the lower frequencies.
It can be due to the following facts: (i) electrons that are travel along the open magnetic field lines diverge with the
radially increasing distance, therefore duration of the radio bursts increases resulting lower drift rates, and (ii)
faster drift rates in the deep solar corona is due to rapid change of frequency of the background
plasma with the increasing distance \citep{Reid2014} .
We have studied the duration of Type IIIg bursts and found that $95.5\%$ of them are lasted within $4$ minutes and they are independent
of lower and upper frequency cut-offs. \\

The meter wave solar radio bursts are launched in the same layer of the solar atmosphere where geo-effective disturbances initiate and
hence, they can be potential signatures to forecast the space weather hazards.
In this study we show two of such events that have caused radio blackouts near Earth.
As previously mentioned, type III bursts are signatures of solar flares and Type II and Type IV bursts are the signatures of CMEs. The enhanced radiation at X ray and EUV wavelengths along with the energetic particles reach the Earth
within few hours and influences the ionosphere and impacts the HF communications.
Note that it takes 1-5 days for a CME to impact the earth depending on the speed and direction.
Ground based e-CALLISTO network with more than 152 stations located at various longitudes is capable of observing solar radio transient
emissions 24 hours a day. Furthermore, different types of radio bursts are associated with various solar transients and therefore, they can play a crucial role in predicting the space weather.

\acknowledgments
This work was supported by International Science Programme (ISP) through Rwanda Astrophysics, Space and Climate Science
Research Group (RASCSRG).
We thank FHNW, Institute for Data Science in Brugg/Windisch, Switzerland for hosting the e-Callisto network; SOHO/LASCO; NOAA; GOES;
SWPC and WDC-SILSO, Royal Observatory of Belgium, Brussels to make their data available online.
The author (C. Monstein) thanks the ISSI - Bern International Team of ‘Why Ionospheric Dynamics and Structure Behave Differently in
The African Sector’ (the team leaders E. Yizengaw \& K. Groves) for valuable discussions about part of the results that are included
in this paper.
We thank both the referees for providing useful and encouraging comments and suggestions which helped in improving the manuscript.

\bibliographystyle{aasjournal