\section{Introduction}
The Kerr spacetime~\cite{Kerr,Kerr-Texas,kerr-intro,kerr-book,kerr-book-2} is perhaps the single most astrophysically important and pre-eminent of the currently known exact solutions to the Einstein field equations.
(For general background discussion see textbooks such as~\cite{Adler-Bazin-Schiffer,D'Inverno,Hartle,Carroll,wald,weinberg, Hobson, MTW}.)
In view of this central physical importance of the Kerr spacetime, it is well worth developing as many distinct viewpoints~\cite{river}, alternative presentations~\cite{doran,natario,rain,Gordon-form,ansatz}, and tractable approximations~\cite{Lense-Thirring,Pfister,4-of-us} to the Kerr spacetime as possible.
We have recently considered various unit-lapse ``rain'' representations of Kerr~\cite{rain}, and now wish to address the interplay between unit-lapse and possible diagonalization of the spatial 3-metric.

It is a classic mathematical result, typically attributed to Darboux~\cite{Darboux:1898}, that under mild conditions 3-manifolds can have their metrics recast in diagonal form~\cite{Darboux:1925,Darboux:1984,Darboux:1991,Darboux:1992,Darboux:2013,Grant:2009}. The question we wish to raise is whether Darboux diagonalization of the spatial slices of the Kerr spacetime is compatible with simultaneously maintaining the unit lapse condition and axial symmetry.
When starting the calculation there were both reasons for optimism and reasons for caution.
On the one hand:
\begin{itemize}
\item
Diagonalization of the 3-metric only imposes 3 coordinate conditions on the 4-metric, while the unit lapse condition only adds 1 more. Imposing 4 coordinate conditions in (3+1) spacetime naively seems plausible.
\item
Boyer--Lindquist coordinates for Kerr explicitly diagonalize the 3-metric, though they are not unit-lapse coordinates.
\item
Many unit-lapse coordinates for Kerr are known to exist~\cite{doran,natario,rain},
including unit-lapse coordinates very closely related to Boyer--Lindquist coordinates~\cite{rain}.
\item
The Lense--Thirring \emph{approximation} to Kerr can explicitly be cast in unit-lapse flat 3-space form~\cite{4-of-us}.
\end{itemize}
On the other hand, the Kerr spacetime sometimes exhibits (perhaps unexpected) obstructions to otherwise plausible conjectures.
For instance, all static spherically symmetric (and some stationary axisymmetric, \emph{e.g.}, Lense--Thirring) spacetimes can be put into Painlev\'e--Gullstrand~\cite{painleve1,painleve2,gullstrand} form, with a flat spatial 3-metric~\cite{4-of-us,poisson}. But not Kerr. Indeed, the spatial 3-slices of Kerr cannot even be put in conformally flat form~\cite{ValienteKroon:2003,ValienteKroon:2004}.
\enlargethispage{10pt}

Unfortunately in the situation that we are interested in, the Kerr spacetime again is problematic ---
we shall demonstrate that in the Kerr spacetime the unit-lapse condition is incompatible with diagonalizing the 3-metric while maintaining axial symmetry. This is more than just an effect of incompatible symmetries, (for instance, things do work nicely for Lense--Thirring spacetime), the incompatibility depends on specific properties of the metric coefficients, and so on specific dynamical features arising from solving the Einstein equations.

\section{Framework and Setup}

All 3 of the fully specific unit-lapse versions of Kerr that we explored in reference~\cite{rain}, (BL-rain, EF-rain, Doran) have spatial 3-metrics of the form:
\begin{equation}
ds^2 = g_{rr}(r,\theta)\,dr^2 +  g_{\theta\theta}(r,\theta)\,d\theta^2 + g_{\phi\phi}(r,\theta)\,d\phi^2
+ 2 g_{r\phi} (r,\theta) \, dr d\phi.
\end{equation}
That is, the pattern of non-zero elements in the spatial 3-metric is:
\begin{equation}
g_{ij} = \left[\begin{array}{ccc} *&0&*\\ 0 &*&0\\ {}*&0&* \end{array}\right]_{ij}.
\end{equation}
We wish to diagonalize this 3-metric, \emph{while preserving unit lapse and manifest axial symmetry}.
To preserve unit lapse, do not touch the time coordinate~\cite{rain}.
To preserve axial symmetry, transformations of the $\phi$ coordinate are limited to
\begin{equation}
\phi=\bar\phi+\Phi(\bar r, \bar\theta); \qquad
d\phi = d\bar\phi +  \Phi_{\bar r} \;d\bar r + \Phi_{\bar \theta} \;d\bar \theta.
\end{equation}
In principle we could do arbitrary things to $r$ and $\theta$:
\begin{equation}
(r,\theta) \to (\bar r, \bar \theta).
\end{equation}
But let us take the analysis one step at a time.

\section{Azimuth-only coordinate transformations}
\label{S:azimuth}

Let us temporarily agree to leave the $r$ and $\theta$ coordinates untouched; only adjust $\phi$.
That is, we consider:
\begin{equation}
r = \bar r; \qquad \theta = \bar\theta; \qquad \phi=\bar\phi+\Phi(\bar r, \bar\theta).
\end{equation}
Thence
\begin{equation}
dr = d\bar r;  \qquad
d\theta = d\bar \theta; \qquad
d\phi = d\bar\phi +  \Phi_{\bar r} \,d\bar r + \Phi_{\bar \theta} \,d\bar \theta.
\end{equation}
So for the 3-metric
\begin{eqnarray}
ds^2 &=& g_{rr}(\bar r, \bar\theta) \,d\bar r^2
+ g_{\theta\theta}(\bar r, \bar\theta)\,d\bar \theta^2
\nonumber\\
&+&  g_{\phi\phi}(\bar r, \bar\theta)\;
 (d\bar\phi + \Phi_{\bar r} \,d\bar r + \Phi_{\bar \theta} \,d\bar \theta)^2
 \nonumber\\
&+&  2 g_{r\phi}(\bar r, \bar\theta)\;
d\bar r
(d\bar\phi + \Phi_{\bar r} \,d\bar r + \Phi_{\bar \theta} \,d\bar \theta).
\end{eqnarray}

Now just pick off the 3 off-diagonal components:
\begin{eqnarray}
g_{\bar r \bar\phi} &=& g_{r\phi}(\bar r,\bar \theta)+ g_{\phi\phi}(\bar r,\bar \theta)\,\Phi_{\bar r};
\\
g_{\bar \theta \bar\phi} &=&  g_{\phi\phi}(\bar r,\bar \theta)\,\Phi_{\bar \theta};
\\
g_{\bar r\bar \theta} &=&
 \Phi_{\bar\theta}\left( g_{\phi\phi}(\bar r,\bar \theta)\,\Phi_{\bar r} + g_{r\phi}(\bar r,\bar \theta) \right).
\end{eqnarray}
We want all 3 of $g_{\bar r \bar\phi} $, $g_{\bar \theta \bar\phi}$, and $g_{\bar r\bar \theta}$ to vanish.
That is, we demand that the 3 PDEs below must be satisfied:
\begin{eqnarray}
E1:&&\qquad\qquad g_{r\phi}(\bar r,\bar \theta)+ g_{\phi\phi}(\bar r,\bar \theta)\,\Phi_{\bar r} =0;
\\
E2:&&\qquad\qquad
g_{\phi\phi}(\bar r,\bar \theta)\,\Phi_{\bar \theta}=0;
\\
E3:&&\qquad\qquad
\Phi_{\bar\theta}\left( g_{\phi\phi}(\bar r,\bar \theta)\,\Phi_{\bar r} + g_{r\phi}(\bar r,\bar \theta) \right)= 0.
 \end{eqnarray}
But, since $g_{\phi\phi}\neq 0$, $E2$ implies $\Phi_{\bar \theta}=0$, which then automatically satisfies $E3$.

But this also implies $\Phi(\bar r,\bar\theta)\to \Phi(\bar r)$, and $E1$ becomes:
\begin{equation}
{d\Phi(\bar r)\over d\bar r} =
- \; {g_{r\phi}(\bar r,\bar\theta)\over  g_{\phi\phi}(\bar r,\bar \theta)}
\end{equation}
This equation is consistent iff
\begin{equation}
\partial_{\bar\theta} \left(  {g_{r\phi}(\bar r,\bar\theta)\over  g_{\phi\phi}(\bar r,\bar \theta)} \right) = 0.
\end{equation}
But inspection of the BL-rain, EF-rain, or Doran unit-lapse version of Kerr shows that this consistency condition is \emph{not} satisfied. (See the appendix for more details.)
Specifically, (recalling that in this section $\bar r=r$ and $\bar\theta=\theta$), we would want to have
\begin{equation}
\partial_{\theta} \left(  {g_{r\phi}(r,\theta)\over  g_{\phi\phi}(r,\theta)} \right) = 0.
\end{equation}
However
\begin{eqnarray}
\left(  {g_{r\phi}(r,\theta)\over  g_{\phi\phi}(r,\theta)} \right)_{{\hbox{\tiny BL-rain}}}
&=&{2mar\sqrt{2mr(r^2+a^2)}\over\Delta}\; {1\over\rho^2\Sigma} ;
\\
\left(  {g_{r\phi}(r,\theta)\over  g_{\phi\phi}(r,\theta)} \right)_{{\hbox{\tiny EF-rain}}}
&=& -  {a \over\Sigma} \left( 1 +{2mr/\rho^2} + \sqrt{2mr/(r^2+a^2)}
\over 1 + \sqrt{2mr/( r^2+a^2)}\right);
\\
\left(  {g_{r\phi}(r,\theta)\over  g_{\phi\phi}(r,\theta)} \right)_{{\hbox{\tiny Doran}}}
&=&  -{a\over\Sigma} \sqrt{2mr\over a^2+r^2}.
\end{eqnarray}
\leftline{Here $\rho^2=r^2+a^2\cos^2\theta$, and $\Delta=r^2+a^2-2mr$, while $\Sigma = r^2+a^2 + {2mra^2\over\rho^2} \sin^2\theta$. }
It is the explicit presence of $\theta$ inside $\Sigma$ that is the obstruction to satisfying the consistency condition.
So no azimuth-only coordinate transformation is capable of diagonalizing the spatial 3-metric.
(We have been rather slow and careful with this calculation to make the general pattern clear.)

\section{Polar-only $(r,\theta)$ coordinate transformations}
\enlargethispage{20pt}

Now let us leave the azimuthal coordinate intact, and only transform the  $(r,\theta)$ plane.
That is, we consider:
\begin{equation}
r = G(\bar r, \bar\theta); \qquad \theta = H(\bar r, \bar\theta); \qquad \phi=\bar\phi.
\end{equation}
Thence
\begin{equation}
dr = G_{\bar r} \;d\bar r + G_{\bar \theta} \;d\bar \theta;  \qquad
d\theta = H_{\bar r} \; d\bar r + H_{\bar \theta}\;  d\bar \theta; \qquad
d\phi = d\bar\phi.
\end{equation}
So for the 3-metric
\begin{eqnarray}
ds^2 &=& g_{rr}(G(\bar r, \bar\theta), H(\bar r, \bar\theta))\,
 (G_{\bar r} \,d\bar r + G_{\bar \theta} \,d\bar \theta)^2
\nonumber\\
&+&  g_{\theta\theta}(G(\bar r, \bar\theta), H(\bar r, \bar\theta))\,
 (H_{\bar r} \,d\bar r + H_{\bar \theta} \,d\bar \theta)^2
\nonumber\\
&+&  g_{\phi\phi}(G(\bar r, \bar\theta), H(\bar r, \bar\theta))\,
 d\bar\phi^2
 \nonumber\\
&+&  2 g_{r\phi}(G(\bar r, \bar\theta), H(\bar r, \bar\theta))\,
(G_{\bar r} \,d\bar r + G_{\bar \theta} \,d\bar \theta) \,d\bar\phi .
\end{eqnarray}

Now to simplify the notation write $g_{ij}(G(\bar r, \bar\theta), H(\bar r, \bar\theta)) \to g_{ij}(\bar r,\bar \theta)$ and then just pick off the 3 off-diagonal components:
\begin{eqnarray}
g_{\bar r \bar\phi} &=&g_{r\phi}(\bar r,\bar \theta)\,G_{\bar r};
\\
g_{\bar \theta \bar\phi} &=&g_{r\phi}(\bar r,\bar \theta) \,G_{\bar \theta};
\\
g_{\bar r\bar \theta} &=&
g_{rr}(\bar r,\bar \theta) \,G_{\bar\theta}  G_{\bar r}
+ g_{\theta\theta}(\bar r,\bar \theta) \, H_{\bar r} H_{\bar\theta}.
\end{eqnarray}

Now we want all 3 of $g_{\bar r \bar\phi} $, $g_{\bar \theta \bar\phi}$, and $g_{\bar r\bar \theta}$ to vanish.

That is we demand that the 3 PDEs below must be satisfied:
\begin{eqnarray}
E1: &&\qquad\qquad
g_{r\phi}(\bar r,\bar \theta)\,G_{\bar r}=0;
\\
E2:&&\qquad\qquad
g_{r\phi}(\bar r,\bar \theta) \,G_{\bar \theta}=0;
\\
E3:&&\qquad\qquad
g_{rr}(\bar r,\bar \theta)\, G_{\bar\theta} G_{\bar r}
+ g_{\theta\theta}(\bar r,\bar \theta) \,H_{\bar r} H_{\bar\theta} = 0.
\end{eqnarray}
But, since we know $g_{r\phi}\neq 0$, then $E1$ implies $G_{\bar r}=0$, and
$E2$ implies  $G_{\bar \theta}=0$.
But then $G(\bar r,\bar\theta)$ is a constant;
so it is not a good coordinate --- we have an inconsistency.
(We do not even need to look at equation $E3$.)
So we cannot diagonalize the spatial 3-metric using only polar $(r,\theta)$ coordinate transformations.

\section{Axisymmetry preserving coordinate transformations}

Now consider the general case --- this is the most general thing one can do \emph{without damaging  the manifest axial symmetry}.
Consider:
\begin{equation}
r = G(\bar r, \bar\theta); \qquad \theta = H(\bar r, \bar\theta); \qquad \phi=\bar\phi+\Phi(\bar r, \bar\theta).
\end{equation}
Thence
\begin{equation}
dr = G_{\bar r} \;d\bar r + G_{\bar \theta} \;d\bar \theta;  \qquad
d\theta = H_{\bar r} \;d\bar r + H_{\bar \theta} \;d\bar \theta; \qquad
d\phi = d\bar\phi +  \Phi_{\bar r} \;d\bar r + \Phi_{\bar \theta} \;d\bar \theta.
\end{equation}
So for the 3-metric
\begin{eqnarray}
ds^2 &=& g_{rr}(G(\bar r, \bar\theta), H(\bar r, \bar\theta))\;
 (G_{\bar r} \,d\bar r + G_{\bar \theta} \,d\bar \theta)^2
\nonumber\\
&+&  g_{\theta\theta}(G(\bar r, \bar\theta), H(\bar r, \bar\theta))\;
 (H_{\bar r} \,d\bar r + H_{\bar \theta} \,d\bar \theta)^2
\nonumber\\
&+&  g_{\phi\phi}(G(\bar r, \bar\theta), H(\bar r, \bar\theta))\;
 (d\bar\phi + \Phi_{\bar r} \,d\bar r + \Phi_{\bar \theta}\, d\bar \theta)^2
 \nonumber\\
&+&  2g_{r\phi}(G(\bar r, \bar\theta), H(\bar r, \bar\theta))\;
(G_{\bar r} \,d\bar r + G_{\bar \theta} \,d\bar \theta)\;
(d\bar\phi + \Phi_{\bar r} \,d\bar r + \Phi_{\bar \theta} \,d\bar \theta).
\end{eqnarray}

Now to simplify the notation write $g_{ij}(G(\bar r, \bar\theta), H(\bar r, \bar\theta)) \to g_{ij}(\bar r,\bar \theta)$. That is
\begin{eqnarray}
ds^2 &=& g_{rr}(\bar r, \bar\theta)\;
 (G_{\bar r} \,d\bar r + G_{\bar \theta} \,d\bar \theta)^2
\nonumber\\
&+&  g_{\theta\theta}(\bar r, \bar\theta)\;
 (H_{\bar r} \,d\bar r + H_{\bar \theta} \,d\bar \theta)^2
\nonumber\\
&+&  g_{\phi\phi}(\bar r, \bar\theta)\;
 (d\bar\phi + \Phi_{\bar r} \,d\bar r + \Phi_{\bar \theta} \,d\bar \theta)^2
 \nonumber\\
&+& 2\, g_{r\phi}(\bar r, \bar\theta)\;
(G_{\bar r} \,d\bar r + G_{\bar \theta} \,d\bar \theta)\;
(d\bar\phi + \Phi_{\bar r} \,d\bar r + \Phi_{\bar \theta} \,d\bar \theta).
\end{eqnarray}
Now just pick off the 3 off-diagonal components:
\begin{eqnarray}
g_{\bar r \bar\phi} &=&g_{r\phi}(\bar r,\bar \theta) \,G_{\bar r}
+ g_{\phi\phi}(\bar r,\bar \theta)\,\Phi_{\bar r};
\\
g_{\bar \theta \bar\phi} &=&g_{r\phi}(\bar r,\bar \theta) \,G_{\bar \theta}
+ g_{\phi\phi}(\bar r,\bar \theta)\,\Phi_{\bar \theta};
\\
g_{\bar r\bar \theta} &=&
g_{rr}(\bar r,\bar \theta) \,G_{\bar\theta}G_{\bar r}
+ g_{\theta\theta}(\bar r,\bar \theta) \,H_{\bar r} H_{\bar\theta}
+ g_{\phi\phi}(\bar r,\bar \theta)\,\Phi_{\bar r}\Phi_{\bar\theta}
\nonumber\\ &&
+ g_{r\phi}(\bar r,\bar \theta) \left[
\Phi_{\bar r} \,G_{\bar \theta}
+\Phi_{\bar \theta} \,G_{\bar r} \right].
\end{eqnarray}
We want all 3 of $g_{\bar r \bar\phi} $, $g_{\bar \theta \bar\phi}$, and $g_{\bar r\bar \theta}$ to vanish.
That is we demand that the 3 PDEs below must be satisfied:
\begin{eqnarray}
E1:&&\qquad\qquad
  g_{r\phi}(\bar r,\bar \theta)\,G_{\bar r}
+ g_{\phi\phi}(\bar r,\bar \theta)\,\Phi_{\bar r}=0;
\\
E2:&&\qquad\qquad
 g_{r\phi}(\bar r,\bar \theta) \,G_{\bar \theta}
+ g_{\phi\phi}(\bar r,\bar \theta)\,\Phi_{\bar \theta}=0;
\\
E3:&&\qquad\qquad
g_{rr}(\bar r,\bar \theta) \,G_{\bar\theta}G_{\bar r}
+ g_{\theta\theta}(\bar r,\bar \theta) \,H_{\bar r} H_{\bar\theta}
+ g_{\phi\phi}(\bar r,\bar \theta)\,\Phi_{\bar r}\Phi_{\bar\theta}
\nonumber\\ &&
\qquad\qquad\qquad\qquad
+ g_{r\phi}(\bar r,\bar \theta) \left[
\Phi_{\bar r} \,G_{\bar \theta}
+\Phi_{\bar \theta} \, G_{\bar r} \right]=0.
\end{eqnarray}
These are 3 PDEs for 3 unknown functions. This is not, by itself,  necessarily problematic.
However, consider the specific linear combination
\begin{equation}
 {1\over g_{\phi\phi}(\bar r,\bar\theta)}\; (G_{\bar\theta}\;E1-G_{\bar r}\;E2).
\end{equation}
Then
\begin{equation}
G_{\bar\theta} \,\Phi_{\bar r}- G_{\bar r}\,\Phi_{\bar\theta} = 0.
\end{equation}
This means that the cross product vanishes:
\begin{equation}
(\Phi_{\bar r}, \Phi_{\bar\theta}) \times (G_{\bar r}, G_{\bar\theta}) = 0.
\end{equation}
This implies
\begin{equation}
(\Phi_{\bar r}, \Phi_{\bar\theta}) \propto (G_{\bar r}, G_{\bar\theta}).
\end{equation}
With general solution
\begin{equation}
\Phi(\bar r, \bar\theta) = W( G(\bar r, \bar\theta)),
\end{equation}
for some arbitrary function $W(G)$.

Now substitute this back into $E1$ and $E2$:
\begin{equation}
E1':\qquad\qquad
G_{\bar r}(\bar r, \bar\theta) \left[ g_{r\phi}(\bar r, \bar\theta)
+ g_{\phi\phi}(\bar r, \bar\theta) \,W'(G(\bar r, \bar\theta)) \right]=0.
\end{equation}
\begin{equation}
E2':\qquad\qquad
G_{\bar \theta}(\bar r, \bar\theta) \left[ g_{r\phi}(\bar r, \bar\theta)
+ g_{\phi\phi}(\bar r, \bar\theta) \,W'(G(\bar r, \bar\theta)) \right]=0.
\end{equation}
Now we cannot have \emph{both} $G_{\bar r}=0$ and $G_{\bar \theta}=0$, since that would mean $G$ is a constant, and so not a good coordinate. Therefore we must have
\begin{equation}
\left[ g_{r\phi}(\bar r, \bar\theta) + g_{\phi\phi}(\bar r, \bar\theta) \, W'(G(\bar r, \bar\theta)) \right]=0.
\end{equation}
That is:
\begin{equation}
  W'(G(\bar r, \bar\theta)) = - { g_{r\phi}( \bar r, \bar \theta) \over  g_{\phi\phi}( \bar r, \bar \theta) }.
\end{equation}
But unwrapping this in terms of the \emph{original} coordinate system we started with, this means we are demanding
\begin{equation}
  W'(r) = - { g_{r\phi}( r, \theta) \over  g_{\phi\phi}( r, \theta) }.
\end{equation}
But inspection of the BL-rain, EF-rain, or Doran unit-lapse versions of Kerr shows that this consistency condition is not satisfied. (The right hand side is explicitly $\theta$ dependent.)
We have
\begin{eqnarray}
\left(  {g_{r\phi}(r,\theta)\over  g_{\phi\phi}(r,\theta)} \right)_{{\hbox{\tiny BL-rain}}}
&=&{2mar\sqrt{2mr(r^2+a^2)}\over\Delta}\; {1\over\rho^2\Sigma} ;
\\
\left(  {g_{r\phi}(r,\theta)\over  g_{\phi\phi}(r,\theta)} \right)_{{\hbox{\tiny EF-rain}}}
&=& -  {a \over\Sigma} \left( 1 +{2mr/\rho^2} + \sqrt{2mr/(r^2+a^2)}
\over 1 + \sqrt{2mr/( r^2+a^2)}\right);
\\
\left(  {g_{r\phi}(r,\theta)\over  g_{\phi\phi}(r,\theta)} \right)_{{\hbox{\tiny Doran}}}
&=&  -{a\over\Sigma} \sqrt{2mr\over a^2+r^2}.
\end{eqnarray}
The $\theta$ dependence is hiding in $\Sigma$.
(See appendix for details on these three metrics. Note the very strong similarities to the azimuth-only argument in section \ref{S:azimuth} above.) So no axisymmetry-preserving coordinate transformation is capable of diagonalizing the spatial metric.

Note that knowing the explicit forms of $g_{r\phi}( r, \theta)$ and   $g_{\phi\phi}( r, \theta)$ in \emph{any one of} the three BL-rain, EF-rain, or Doran unit-lapse versions of Kerr is enough to get to this conclusion.
Also note that we never had to use equation $E3$.
This completes the argument --- in the unit-lapse Kerr context the Darboux diagonalization argument is incompatible with manifest axisymmetry.

\section{Discussion and Conclusions}

What have we learned from this discussion? On the one hand, this no-go theorem is a very specific mathematical result specifically for the Kerr spacetime. On the other hand, the various ingredients that go into the discussion have a very much wider realm of applicability. For instance,
 the unit-lapse spacetimes occur quite commonly and very naturally in many specific examples of analogue spacetimes~\cite{unexpected, visser:1997, visser:1998, volovik:1999, stone:2001, visser:2001, fischer:2002, novello:2002, probing, Visser:vortex, LRR, Liberati:2005, Weinfurtner:2005, visser:2010, Visser:2013, Liberati:2018, Schuster:2018}.  In an analogue spacetime context the unit lapse condition physically corresponds to a constant signal propagation speed. (This holds, for example, to a good approximation for sound waves in water.) Various analogue spacetimes can then be invoked to help one develop physical intuition in the current more purely general relativistic context, directly relevant to modelling infall and accretion.

Astrophysically, unit lapse versions of stationary spacetimes, (even if they are not vacuum solutions of the Einstein equations), are extremely useful in at they immediately provide an enormous class of timelike geodesics, the ``rain geodesics''. These correspond to the zero angular momentum observers, ZAMOs, that are dropped from spatial infinity with zero initial velocity and zero angular momentum. These rain geodesics  provide an explicit  and quite tractable probe of the spacetime physics.

The underlying theme behind our attempt at imposing Darboux diagonalization was to continue the search for improved coordinate systems for the Kerr spacetime. Finding such improved coordinate systems (if possible) is  strategically and tactically important for a better understanding of the technically quite challenging and astrophysically important Kerr spacetime; see  particularly the discussion in reference~\cite{kerr-intro}. See also, for instance, recent attempts at finding a ``Gordon form'' for the Kerr spacetime~\cite{Gordon-form}, and recent attempts at upgrading the ``Newman--Janis trick'' from an ansatz to an algorithm~\cite{ansatz}.

Finally, we should also emphasise that the discussion herein also impacts on and informs the discussion and investigation regarding the potential observational ability to distinguish exact Kerr black holes from various ``black hole mimickers'' that have been suggested in the literature --- see for instance references~\cite{small-dark-heavy, BH-in-GR}.  More recently one could consider references~\cite{phenomenology, viability, geodesicaly-complete, pandora, causal, LISA}, and references~\cite{Simpson:2020, Simpson:2019, Simpson:2018, Boonserm:2018, Simpson:2019-core,
Berry:2020-core-1, Berry:2020-core-2, Bardeen:1968,Hayward:2005,Frolov:2014,Frolov:2017}.

\newpage


\section{Discussion}\label{sec:discussion}
Accessibility is a key to wider adoption of any technology and machine learning is no exception. Here  we introduced  data programming by demonstration (DPBD), a general human-in-the-loop framework that aims to ease writing labeling functions, improving the accessibility and efficiency of data programming.  We then presented \system, a DPBD system, for easily generating labeling functions to create training datasets for document-level classification tasks. \system converts user rationales interactively expressed as span-level annotations and relations among them to labeling rules using the DPBD framework. We also reported our progress in developing \tagruler, a second DPBD system focusing on labeling functions for tagging.

Through a user study with 10 data scientists performing real world labeling tasks for classification, we evaluated \system together with conventional data programming and found that \system enables more accessible data programming without loss in the performance of labeling models created. We believe DPBD systems will be useful for data scientists as well as subject matter experts. We release \system as open source software to support future applications and extended research.

\section{Evaluation}\label{sec:exp}
We evaluate \system alongside manual data programming using Snorkel~\cite{ratner2017snorkel}. Our goal is to better understand the trade-offs afforded by each method. To this end, we conducted a user study with data scientists and measured  their task performance accuracy in completing two labeling tasks. In addition to task performance, we also analyzed the accessibility and  expressivity of both methods using the qualitative feedback elicited from participants.

Note that \system can be used by programmers and non-programmer domain experts alike, but a fair comparison with Snorkel requires proficiency in conventional programming.

\subhead{Participants}  We recruited 10 participants with Python programming experience through our professional network. All participants had significant programming experience (avg=$12.1$ years, std=$6.5$). Their experience with Python programming ranged from $2$ to $10$ years with an average of $5.2$ years (std=$2.8$).

\subhead{Experimental Design}  We carried out the study using  a within-subjects experiment design, where all participants performed tasks using both conditions (tools).  The sole independent variable controlled was the method of creating labeling functions. We counterbalanced the order in which the tools were used, as well as which classification task we performed with which tool.

\subhead{Tasks and Procedure} We asked participants to write  labeling functions for two prevalent labeling tasks: spam detection and sentiment classification.  They performed these two tasks on  YouTube Comments and Amazon Reviews, respectively. Participants received 15 mins of instruction on how to use each tool, using a topic classification task (electronics vs. guns) over a newsgroup dataset~\cite{rennie200820} as an example. We asked participants to write as many functions as they considered necessary for the goal of the task.  There were given 30 mins to complete each task and we recorded the labeling functions they created and these functions' individual and aggregate performances.  After completing both tasks, participants also filled out an exit survey, providing their qualitative feedback.

For the manual programming condition, we provided a Jupyter notebook interface based on the Snorkel tutorial. The notebook had a section for writing functions, a section with diverse analysis tools, and a section to train a logistic regression model on the labels generated.

\section{DPBD Framework}\label{sec:framework}
\stitle{Problem Statement}
Given a dataset $D=\set{d_1, \ldots, d_m}$ of data records and a set of labels $L=\set{l_1,\dots, l_n }$,  we aim to develop a framework that enables human labelers to interactively assign a label from $L$ for each data record efficiently sampled from  $D^\prime \subset D$ ($|D^\prime| \ll |D|$), while demonstrating their rationales for label assignments through visual interaction. Given a triplet $(d_i^\prime,  v_i, l_j)$ of a data record, a visual interaction from the labeler, and the label assigned,  we want this framework to effectively synthesize and propose labeling rules $R_{ij} = \set{r_1,\ldots,r_k}$ for the labeler to choose from.
Finally, we want the framework to optimally aggregate all the chosen rules (labeling functions) in order to create a labeled training set from  $D\setminus D^\prime$ with probabilistic labels in order to subsequently train discriminative models on it.

\stitle{Framework Overview}
The data programming by demonstration (DPBD) framework  (Figure~\ref{fig:overview}) has two input sources: the human \emph{labeler}, and the \emph{data} that is to be labeled. The labeler is the subject matter expert who has sufficient domain understanding to extract useful signals from data. Given a dataset, our framework enables the labeler to label each record with a categorical label, while providing their labeling rationales by interactively marking relevant parts of the record and specifying semantics and relationships among them.
The output is a labeling model, which is trained to automatically produce labels for the large set of unlabeled data.
The DPBD framework has four main components, \emph{labeling interface},
\emph{synthesizer}, \emph{modeler}, and \emph{active sampler}.

\subsection{Labeling Interface}\label{sec:interface} The labeling interface is the workplace where the labeler encodes domain knowledge into labeling rules. It provides a way to express noisy explanations for labeling decisions using a visual interaction language,  which allows the user to express domain knowledge without having to formalize their ideas into computer programs or natural language explanations. This allows for more focus on patterns in the data while abstracting away any implementation concerns.

\stitle{Generalized Labeling Model} Inspired by the entity-relationship model~\cite{DBLP:journals/tods/Chen76} in database modeling, the generalized labeling model (GLM) models the data records with \emph{concepts} and \emph{relationships}.  The GLM views the data record as a series of tokens,
where a token is a continuous subset of a record with no semantics attached.
For example, in text data, a token can be any span (single char to multiple words) of the data record; in an image data record, it would be a 2D region, rectangular or free form; and in an audio data record, it would be a 1D window of the data record (e.g., a phoneme).  A \emph{concept} is a group of tokens that the labeler believes share common semantics. For instance, over text data, the labeler might define a concept of positive adjectives consisting of a set of tokens, each of which can imply a positive review.
When labeling audio data, the labeler might create a concept to aggregate all clips that express excitement, or of a specific speaker.  This abstraction allows the user to teach the GLM which generalizations are relevant to the task.
A \emph{relationship} represents a binary correlation between token-token, token-concept, or concept-concept. Some examples are membership (e.g., a token is in a concept), co-existence (e.g., opinion and aspect tokens), and positional (e.g., a person is standing left to a table~\cite{DBLP:conf/ijcnn/HaldekarGO17}).

\begin{table}[h]
\centering
  \caption{Mapping from GLM elements to operations in the labeling interface.\label{tab:GLM}}
  \begin{tabular}{cc}
    \toprule
    GLM Element & Operations\\
    \midrule
     token & select, assign\_concept \\
     concept & create, add, delete\\
     relationship & link, direct\_to\\
  \bottomrule
\end{tabular}
\end{table}

\stitle{Mapping GLM Elements to Operations}
Given the GLM specification described above, our framework also defines the operations that can be applied on GLM elements. Table~\ref{tab:GLM} lists the GLM elements and the corresponding operations. The implementation of both the labeling interface and the operations described in Table~\ref{tab:GLM} would vary across data types and token definitions. To add expressivity, the GLM may also perform transformations over the set of tokens, as we describe in the next section.

\stitle{Compiling Operations into Labeling Rules}
Once the labeler finishes annotating an example using the provided operations, and selects a label, the tokens are extracted from the annotation and used
as the initial set of conditions from which to build rules.
The synthesizer combines these conditions into labeling rules by selecting subsets of the conditions to be combined with different conjunctive formulas, according to the relationships the user has annotated.
The synthesizer extends the initial set of labeling rules and presents the extended labeling rules for the labeler to select from, choosing desired ones based on  domain knowledge.

A labeling rule serves as an intermediate language, interpretable by both the labeler and the synthesizer. In our framework, we adopt the notation of domain relational calculus~\cite{DBLP:conf/vldb/LacroixP77} to represent these rules, which can be expressed as:
$\{\texttt{tokens} \mid \texttt{conditions}\}\Rightarrow  \texttt{label} $.
The variable \texttt{tokens} is a sequence of tokens with existential quantification, and
 \texttt{conditions} is a conjunctive formula over boolean predicates that is tested over \texttt{tokens} on a data record.

The predicates are first-order expressions, and each can be expressed as a tuple $(T, lhs, op, rhs)$. $T$ is an optional transformation function on a token identifier, a process of mapping the raw token to more generalized forms. Some example transformations are word lemmatization in text labeling, speech-to-text detection in audio labeling, or object recognition in image labeling.
$lhs$ is a token, while $rhs$ is can be either token, literal or a set.
If $rhs$ denotes a token, the transformation function $T$ may also apply to $rhs$.
$op$ is an operator whose type depends on the type of $rhs$.  If $rhs$ is a token or literal,  $op$ detects a positional or an (in)equality relationship. Otherwise, if $rhs$ is a set, $op$ is one of the set operators  $\{\in, \not\in \}$. Since the \texttt{conditions} is in the conjunctive form, the order of labeler's interactions does not matter.

\example{
Consider the following review for the binary sentiment classification (positive or negative) task:

\texttt{This book was so great! I loved and read it so many times that I will soon have to buy a new copy.}

If the labeler thinks this data record has a positive sentiment, she can express her decision rationale using GLM.
First, she may select two tokens that are related to the sentiment: \texttt{book} and \texttt{great}. Assume there are two concepts the labeler previously created: (1) \texttt{item$=\{$book, electronics$\}$}; and
(2) \texttt{padj$=\{$wonderful$\}$}. The labeler realizes the token \texttt{great} can be generalized by the \texttt{padj} concept, which means that the labeling rule will still be valid if this token is replaced by any tokens in the concept, so she adds this token to the concept.

Finally, the labeler creates a positional relationship from \texttt{book} to token \texttt{great} to indicate that they appear in the same sentence, before completing the labeling process.
These operations compile into the labeling rule
$r: \{t_1, t_2 \mid t_1=\texttt{book} \land t_2 \in \texttt{padj} \land idx(t_1) < idx(t_2)  \}\Rightarrow \texttt{positive}$.
}

This rule is sent to the synthesizer for expansion and program synthesis.

\subsection{Synthesizer}\label{sec:synthesizer}
Given the compiled labeling rule from the labeling interface, the synthesizer extends one single labeling rule from labeler's interaction to a set of more general labeling rules; and translates those labeling rules into computer programs.
It is straightforward to translate the rules into executable computer programs (labeling functions), so in this section, we focus on how to synthesize the extended labeling rules.

Given the labeling rule compiled from a labeler's interaction, the synthesizer generates more labeling rules while optimizing two competing goals: maximizing generalization, so that more (unseen) data can be accurately labeled; and maximizing the coverage of the labeler's interaction, simply because labeler's interaction is the most valuable signal for labeling based on the domain knowledge.
Of course, the larger the set of annotations in an interaction, the larger the set of labeling functions that can be synthesized. To keep rule selection as easy as possible for the user, in this case we prioritize rules that cover more of the interaction, assuming that there is little redundancy

We achieve generalization of the given rules using the following heuristics:
(1) substituting tokens with concepts; (2) replacing general coexistence relationships with position-specific ones; and (3) applying the available transformations over the tokens (for example, object recognition in a section of an image).

Once the extended rules are generated, the rules are ranked by their generalization score---a measurement of how applicable a certain rule is.
We define a data-independent generalization score for a labeling rule $r$ as: $ G(r) = \prod_{c\in r.conds} |c.rhs|$.
Intuitively, $G(r)$ is calculated by counting how many different data instances that $r$ can be used.

\example{
Continuing with our Amazon review example, the synthesizer can derive the following labeling rules from $r$ using these heuristics:
\begin{enumerate}
\small{
\item $\{t_1, t_2 \mid t_1 \in \texttt{item} \land t_2 \in \texttt{padj} \} \Rightarrow \texttt{positive}$\label{rule1}
\item $\{t_1, t_2 \mid t_1 \in \texttt{item} \land t_2 \in \texttt{padj} \land idx(t_1)<idx(t_2)\}\Rightarrow \texttt{positive}$\label{rule3}
\item $\{t_1, t_2 \mid t_1=\texttt{book} \land t_2\in \texttt{padj} \}\Rightarrow \texttt{positive}$\label{rule5}
}
\end{enumerate}
Note that labeling rule~(\ref{rule1}) is more general than~(\ref{rule3}) and~(\ref{rule5}) because all data records that can be labeled by~(\ref{rule3}) and~(\ref{rule5}) will be labeled the same way using labeling rule~(\ref{rule1}).
}

The top-k candidates ranked by the generalization score are displayed in the labeling interface for the labeler to accept or reject.

\subsection{Modeler}\label{sec:modeler}
The modeler component trains a model that can be used to automatically annotate unlabeled datasets.
Naively aggregating the labeling functions  can be either inaccurate (since labeling functions can be conflicting and correlated), or does not scale with a large set of unlabeled data~\cite{ratner2017snorkel}. Instead, the modeler encapsulates the ideas from traditional data programming~\cite{ratner2019training, bach2017learning, ratner2017snorkel,ratner2016data} to first build a generative model to denoise labeling functions, and then train a discriminative model to leverage other features beyond what are expressed by the labeling functions.

\subsection{Active Sampler}\label{sec:sampler}
To improve the model quality at faster rates, our framework uses an active sampler to choose the next data record for labeling.  The active sampler can be plugged in with any custom active learning policy.  By default, it selects the data record $x^*$ with the highest entropy (i.e., the one that the labeling model is currently  the most uncertain about): $ x^* = \argmax_x - \sum_i^{|L|} p_\theta(L_i\mid x)\log{p_\theta(L_i\mid x)}$  where $p_\theta(L_i\mid x)$ is the probability that example $x$ belongs to class $L_i$, as predicted by the trained label model.

\section{Introduction\label{sec:intro}}
Machine learning (ML) models used in practice today are predominantly supervised models and rely on large datasets labeled for training. However, the cost of collecting and maintaining labeled training data remains a bottleneck for training high-capacity supervised models~\cite{sculley2015hidden}. Data programming~\cite{bach2017learning,ratner2017snorkel,ratner2016data} aims to address the difficulty of collecting labeled data by using a programmatic approach to weak supervision by heuristics, where domain experts are expected to provide data programs (labeling functions) incorporating their domain knowledge. Prior work on data programming focuses on modeling and aggregating labeling functions written manually~\cite{ratner2017snorkel,ratner2016data} or generated automatically~\cite{hancock2018training, varma2018snuba} to denoise labeling functions.

Writing data programs can be, however, challenging and time consuming.  Most domain experts or lay users have no or little programming literacy, and even for those who are proficient programmers, it is often difficult to convert domain knowledge to a set of rules by writing programs.

\begin{figure}[tbp]
    \centering
    \includegraphics[width=0.9\linewidth]{figures/overview.pdf}
    \caption{Overview of the data programming by demonstration (DPBD) framework. Straight lines indicate the flow of domain knowledge, and dashed lines indicate the flow of data.\vspace{-10pt}\label{fig:overview}}
\end{figure}
To address these challenges, we introduce data programming by demonstration (DPBD),  a new framework that aims to make creating labeling functions  easier by learning them from users' interactive visual demonstrations. DPBD moves the burden of writing labeling functions to an intelligent synthesizer while enabling users to steer the synthesis process at multiple semantic levels, from providing rationales relevant for their labeling choices to interactively filtering the proposed functions. DPBD draws from two lines of prior research; programming by demonstration (PBD) or example (PBE), e.g.,~\cite{gulwani2011asp,lieberman2001your}, which aims to make programming easier by synthesizing them based on user interactions or input and output examples, and  interactive learning from user-provided features or rationales ~\cite{zaidan2008modeling,zaidan2007using}.

We operationalize our framework with \system, an interactive system that enables more accessible data programming to create labeled training datasets for document classification. \system automatically generates  document level labeling rules from  span-level annotations and their relations on specific examples provided by users. Through a user study conducted with  10 data scientists, we evaluate  \system alongside manual data programming using Snorkel~\cite{ratner2017snorkel}. We measure the predictive performances of models created by participants for two  common labeling tasks, sentiment classification and spam detection. We also elicit ratings and qualitative feedback from participants on multiple measures, including  ease of use, ease of learning, expressivity, and overall satisfaction.  We find \system facilitates more accessible creation of labeling functions without a loss in the quality of learned labeling  models.

Tagging or token level classification in text documents is another widely used task that can benefit from DPBD. Here we also briefly discuss our work in progress on \tagruler, a DPBD system that learns token labeling functions through user interaction to create training datasets for tagging models.

In summary, we contribute (1) DPBD, a general data independent framework for learning labeling rules by interactive demonstration; (2) \system, an interactive system operationalizing our framework for document classification tasks; and (3) a comparative user study conducted with data scientists in performing real world tasks to evaluate \system and conventional data programming. We have made our research artifacts, including the \system code and demo, publicly available~\footnote{\url{https://github.com/megagonlabs/ruler}}.

\section{Related Work}\label{sec:related}
We build on prior work on weak supervision, programming by demonstration, and learning from feature annotations by users.

\subhead{Weak Supervision}\label{sec:weak_supervision}
To reduce the cost of labeled data collection, weak supervision methods leverage noisy, limited, or low precision sources such as crowdsourcing~\cite{karger2011iterative}, distant supervision~\cite{mintz2009distant}, and user-defined heuristics~\cite{gupta2014improved} to gather large training data for supervised learning. Data programming~\cite{ratner2017snorkel,ratner2016data} is a programmatic approach to weak supervision using heuristics, where labeling functions provided by domain experts are used to create training data at scale.

\subhead{Program Synthesis by Demonstration}\label{sec:program_synthesis}
Automated synthesis of programs that satisfy a given specification is a classical artificial intelligence (AI) problem~\cite{waldinger1969prow}. Generating programs by examples or demonstration is an instance of this problem. The terms programming by example (PBE), or programming by demonstration (PBD) have often been used interchangeably, though their adoption and exact meaning might diverge across fields and applications.  PBD systems aim to empower  end user programming in order to improve user productivity
~\cite{allen2007plow,leshed2008coscripter,li2017sugilite,myers1998topaz,hanafi2017seer,heer2015predictive,jin2019clx}.
One of the core research questions in PBD is how to generalize from seen examples or demonstrations. To generalize, PBD systems need to resolve the semantic meaning of user actions over relevant (e.g., data) items.  Prior approaches incorporate a spectrum of user involvement, from making no inference (e.g.,~\cite{halbert1993smallstar, myers1998topaz}) to using AI models with no or minimal user involvement, to synthesize a generalized program (e.g.,~\cite{gulwani2011asp,lau2003programming,mcdaniel1999getting,menon2013machine,jin2019clx}). Our framework takes a hybrid approach within the spectrum above and combines inference and statistical ranking along with interactive demonstration.

\subhead{Learning from Feature Annotations}\label{sec:feature_annotation}
Prior  work proposes methods for learning from user provided features~\cite{druck2009active,liang2009learning,raghavan2005interactive}, rationales~\cite{arora2009interactive,von2006peekaboom,zaidan2008modeling,zaidan2007using}, and natural language explanations~\cite{hancock2018training,srivastava2017joint}. BabbleLabble~\cite{hancock2018training} uses a rule-based parser  to turn natural language explanations into labeling functions and aggregates these functions using Snorkel.
\system also learns labeling functions from high level imprecise explanations and aggregates them using the Snorkel framework. However, \system  enables users to supply their rationales through interactive visual demonstrations, reducing the cognitive load to formalize one's intuition.

\begin{figure}[tbp]
\centering
\includegraphics[width=\linewidth]{figures/ruler-vs-snorkel-qual.pdf} \caption{Participants' subjective ratings on ease of use, expressivity, ease of learning and overall satisfaction, on a 5-point Likert scale. \system is considered by participants to easier to use and learn, though using Snorkel alone is considered to be more expressive.\vspace{-10pt} \label{fig:subjectiveratings}}
\end{figure}
\section{Results}\label{sec:results}
To analyze the performance of the labeling functions created
by participants, for each participant and task, we select the labeling
model  that achieved the highest f1 score on the development set.  For each labeling model, we then train a logistic regression model on a training dataset  generated by the model.  We finally evaluate the performance of the logistic regression model on a heldout test set (400 examples). We also analyze the subjective ratings provided by participants on a Likert scale of 5 (1--5, higher is better)  in their exit surveys.
We use the paired Wilcoxon signed rank test to assess the significance of differences in prediction metrics and subjective ratings between \system and Snorkel. We also report the effect size $r$ for all our statistical comparisons.

\subhead{Model Performance}
We find that \system and Snorkel provide comparable model performances (Figure~\ref{fig:modelperfm}). The logistic regression models trained on data produced by labeling models created using \system have slightly higher f1 ($W=35$, $p=0.49$, $r=0.24$ ), precision ($W=30$, $p=0.85$, $r=0.08$), and recall ($W=25$, $p=0.85$, $r=0.08$) scores on average. Conversely, accuracy is slightly higher ($W=17$, $p=0.32$, $r=0.15$) for Snorkel models on average than \system. However these differences are not statistically significant.

\subhead{Subjective Ratings and Preferences}
Participants find \system to be significantly easier to use
($W=34$, $p=0.03 < 0.05$, $r=0.72$) than Snorkel (Figure~\ref{fig:subjectiveratings}).
Similarly, they consider \system easier to learn ($W=30$, $p=0.1$, $r=0.59$) than Snorkel.  On the other hand, as we expected, participants report Snorkel to be more expressive ($W=0$, $p=0.05$, $r=0.70$)  than  \system. However, our participants appear to consider accessibility (ease of use and ease of learning) to be more important criteria, rating \system higher ($W=43$, $p=0.12$, $r=0.51$) than Snorkel for overall satisfaction.
These results show that \system is more accessible and offers better overall experience, while providing model performance comparable to Snorkel. \system can help data scientists save time and create better models, either in conjunction with traditional programming or alone. For the user who is not skilled at programming, \system is also the only tool available to help leverage data programming with control over the functions.

\section{Ruler}\label{sec:ruler}
\system is an  interactive system that implements the data programming by demonstration (DPBD) framework to  facilitate labeled training data preparation for document-level text classification models. For this, \system leverages   span-level features and relations in text documents demonstrated  through visual interactions by users (labelers). To begin a labeling task, the data owner needs to upload their unlabelled dataset, in addition to a small labeled development set, and optionally a small test and validation set.  This mirrors the data requirements of Snorkel,  which the underlying DPBD modeler encapsulates. We next discuss the user interface and interactions of \system along with its implementation details in operationalizing DPBD for text.

\subsection{User Interface and Interactions}
Recall that the purpose of the labeling interface in DPBD ( Section~\ref{sec:interface}) is to enable the labeler to encode domain knowledge into rules through visual interaction. To this end, \system interface provides affordances through 6 basic views (Figure~\ref{fig:ruler}), which we briefly describe below---the letters A-F refer to annotations in Figure~\ref{fig:ruler}.

\begin{figure}[tbp]
    \centering
    \includegraphics[width=\linewidth]{figures/labeling_pane.png}

    \caption{\system Labeling Pane, where the user conveys domain knowledge using a visual interaction language. Annotations are color coded by the concepts they are assigned to.\vspace{-5pt}\label{fig:labeling_pane}}
\end{figure}

\subhead{Labeling Pane} (A) is the main view where the user interacts with document text. Labeling Pane (Figure~\ref{fig:labeling_pane}) shows  contents of a single document at a time and supports all the labeling operations defined by the GLM in the context of text data.  The user can annotate spans either by highlighting them directly with the cursor or adding them to a concept. These spans can be linked together if the relationship between them is significant to the user. Once the user selects a document label (class)  from the options displayed, the system synthesizes a diverse set of labeling functions to suggest to the user.

\subhead{Concepts Pane} (B) allows users to create concepts, add and edit tokens (whole words surrounded by non-alphabetical characters) or regular expressions, and see annotations over their text automatically added when a match is found (Figure~\ref{fig:concepts_pane}). This interaction allows users to abstract away details about specific language use by grouping tokens or regular expressions into  concepts.

\subhead{Suggested Functions} (C) shows the labeling functions suggested by the system. The user can select any functions that seem reasonable, and only then are they added to the underlying labeling model that is iteratively built.

\subhead{Labeling Statistics} (D) displays current statistics of the label model computed over the development set, and differential changes incurred by the last data interaction. Because this panel updates as the user interacts, they can quickly explore the space of labeling functions with a very low cost in terms of time, computation, and human effort.

\subhead{End-model Statistics} (E) shows the performance statistics for an end model for which the user intends to collect training data. For example, in our user study we used a logistic regression model  with a bag of words features on the generated training data. This model is evaluated on a small held-out test set, and its performance metrics are shown in this pane.

\subhead{Selected Functions} (F) lists of currently selected labeling rules that make up the labeling model along and shows each rule's performance statistics based on the development set. The user can click to open a details panel showing observed incorrect labels and sample texts labeled by this function.

\subsection{Server and Model}
\system's backend comprises the synthesizer (Section~\ref{sec:synthesizer}), modeler (Section~\ref{sec:modeler}) and active sampler (Section~\ref{sec:sampler}) components. The backend components are all implemented in Python 3.6.

In addition to the function generation defined in Section~\ref{sec:synthesizer}, \system's synthesizer also augments labeling rules using existing text processing libraries. It enhances the text with transformations that recognize named entity types such as \texttt{person} and \texttt{location}, extracted using the spaCy library~\cite{spacy}. These annotations are made visible to the user, and annotations containing named entities will generate functions that generalize to all instances of that entity.

In our implementation, relationships can include co-occurrence in the same sentence as well as in the document.
\system encapsulates the Snorkel library~\cite{ratner2017snorkel} into its modeler to aggregate the generated labeling functions.

\begin{figure}[tbp]
    \centering
    \includegraphics[width=\linewidth]{figures/concept_detail.png}

    \caption{Left: Example concept created to capture negation.  Right: example text highlighting as concept elements are matched in the text, and annotations created once the element is submitted.\label{fig:concepts_pane}}
\end{figure}

\section{TagRuler}\label{sec:tagruler}
Another important task relevant to text documents is tagging,
the process of classifying token sequences in documents, e.g., as name, person, organization, aspect, opinion, number, etc. Tagging has many applications, including named entity recognition (NER), information extraction, relation extraction, semantic role labeling, and question answering.  Curating training data for tagging models can be a challenge, particularly in domains where experts are needed for data labeling. Based on the DPBD framework, we have been also developing \tagruler (Figure~\ref{fig:tagruler}), an interactive system for learning labeling functions by demonstration for document tagging. While \tagruler shares the underlying framework and user interface characteristics with \system, it tackles a new set of challenges. First, the possibilities for interactive demonstration, where the labeling task itself involves span highlighting, are limited. This makes both intuitive visual interaction design and effective automated inference based on the context of user demonstrations critical. Second, the space of labeling functions can be very large and therefore having  fast and effective synthesis and ranking is crucial. Third, due to the increased number of labeling functions considered, computational performance to sustain interactivity can be a bottleneck.

\tagruler currently synthesizes two classes of rules based on the semantic and syntactic analysis of the document context. For semantic analysis, we use the pretrained language models BERT~\cite{devlin2019bert} and ELMo~\cite{peters2018elmo}. As for syntactic analysis, we utilize part of speech (POS) and NER tags along with dependency parsing. To aggregate functions synthesized, the system  provides four alternative models; a hidden Markov model~\cite{safranchik2020weakly}, FlyingSquid~\cite{fu2020fast},
Snorkel MeTaL~\cite{ratner2019training}, and a majority voting based model. We are currently evaluating the tradeoffs among these alternative  models in the interactive setting of \tagruler.
\begin{figure}[tbp]
    \centering
    \includegraphics[width=\linewidth]{figures/tagruler.png}
    \caption{ \tagruler is a DPBD system that synthesizes token level labeling functions based on user annotations.
    \label{fig:tagruler} \vspace{-5pt}}
\end{figure}