\section{Introduction \& Related Work}

Thorough documentation of a patient's clinical encounters remains a highly sought goal in modern healthcare systems.
Especially of importance are the patient's initial complaints, which often guide diagnostic measures.
These are usually delineated in a patient monologue during an initial anamnesis interview between physician and patient.
During such conversations, documentation is often recorded in handwritten or typed form. This is very time effective, but usually results in brief, non-standardized, and potentially incomplete notes which are difficult to query or investigate. The professional is often forced to trade off documentation completeness, patient attention and time expenditure.
A system that automatically extracts relevant information from such monologues could greatly improve documentation completeness and time efficiency.

Recent methods achieve promising results for neural abstractive summarization (\citealt{shi2018neural}). However, such sequence to sequence approaches usually require large amounts of training data, which is particularly hard to obtain in the medical domain (\citealt{quiroz2019challenges}).

Other work investigates symptom extraction from clinical text. \cite{jackson2017natural} extract symptoms from discharge notes, and others classify chief complaints from Electronic Health Records (\citealt{osti_1637127}, \citealt{Chapman2005ClassifyingFT}).
In the conversational domain, \cite{liu-etal-2019-fast} identify attributes of symptoms on a corpus of English post-discharge nurse-patient dialogues.

Recently, the transformer-based (\citealt{vaswani2017attention}) BERT model (\citealt{devlin-etal-2019-bert}) attained state of the art results on various NLP tasks. The approach has also been applied in medical informatics, yielding domain-specific models such as Clinical BERT (\citealt{alsentzer-etal-2019-publicly}) that have been successfully applied to medical information extraction (\citealt{10.1093/jamia/ocz096}, \citealt{10.1093/jamiaopen/ooaa022}).
In this paper, we focus on a dataset of German patient monologues. Unfortunately, there are much fewer resources available for the German language; e.g. no domain specific pretrained BERT models exist, and  publicly available medical data is scarce. This significantly increases the difficulty of creating effective models when dealing with German text.

Our goal in this work is to extract all symptoms out of German patient monologues. Our contributions are threefold:
i) we present a dataset that consists of written descriptions of patient conditions, ii) we propose a pragmatically grounded task in which each description is annotated with a structured summary of reported symptoms and their attributes, and iii) we present BERT-based models to solve the task.

\section{Dataset}
We collect publicly accessible text-based patient written condition descriptions (henceforth denoted as ``post'') from German medical forums. We select posts that describe gastrointestinal conditions.

After conducting a survey of active medical professionals in Germany, we find that extracting concise symptom information from the often non-technical and verbose patient descriptions is most relevant in the anamnesis. Although the history of treatment, condition, and medication are also useful, we focus on symptom extraction as a first step.

For each forum post, we label the described symptoms according to a hierarchical collection of common symptoms. Similar to \cite{liu-etal-2019-fast}, we further annotate attributes that convey additional information, categorized as one of several categories: location, description, time, frequency and action. The survey participants confirm that this serves as a useful summary.
We formulate rigorous rules to reduce room for interpretation. To ensure consistency, we label a large portion of the posts twice and merge the results in a final review.

In total, the dataset consists of $125$ unique posts, totaling $592$ symptoms and $1276$ attributes, of which $729$ were unique post segments. Further details and dataset statistics can be found in Appendix \ref{datasetdetails}.
\section{Methods}
To infer the labelings, we divide the task into two stages: \emph{Symptom Classification} and \emph{Attribute Extraction}. In both stages, we make use of a pretrained German BERT model (\citealt{devlin-etal-2019-bert}) from HuggingFace's Transformers (\citealt{Wolf2019HuggingFacesTS}): we use \texttt{bert-base-german-cased} provided by \texttt{deepset.ai} for all experiments.

\subsection{Symptom Classification}
\paragraph{Baseline Models.}
We present two baseline models. The first is an MLP with a sigmoid output layer, trained on TF-IDF (\citealt{Ramos2003UsingTT}) features. The second is a pretrained BERT model followed by a sigmoid classification layer. For both models, the dimension of the output layer corresponds to the number of output symptoms in the training set.
\paragraph{BERT Symptom Query Model.}
The baseline classifiers can naturally only predict symptoms that occur in the training dataset. We propose a symptom query model which is able to generalize to unseen symptoms. We feed the symptom description -- a short, layman description of the symptom -- along with the patient written text into the model, and attempt to predict whether the symptom is present in the text or not.
Due to the large amount of possible negative samples, we randomly select one negative sample for each positive sample in a post.

\paragraph{Curriculum Learning.}
To gain further performance, we employ a curriculum learning scheme (\citealt{bengio2009curriculum}). The hierarchical nature of the symptom collection allows us to form negative samples which are gradually more complex to classify. For instance, symptoms which are close in the hierarchy are more difficult to differentiate.

\paragraph{Augmented Descriptions.}
To improve generalization ability to new symptoms, we extract all text segments in the corpus that were annotated as symptoms and introduce them as alternative layman descriptions of the respective symptom during training.
This approach yields an order of magnitude increase in the amount of positive samples.

\subsection{Attribute Extraction.}
To extract the relevant attributes for a given symptom we adopt a question answering (QA) style approach. The model receives the post and the symptom description as input and predicts which text sections constitute attributes.
Unlike in many QA tasks, (e.g. \citealt{rajpurkar-etal-2016-squad}, \citealt{liu-etal-2019-fast}), we often require multiple answers to a single question, as a symptom can have multiple attributes of the same type.
We introduce two methods to solve this task. For each of the methods, we train six models; a separate model for each of the five attribute types, and a ``general'' model that is trained to predict all attributes at once (see Appendix \ref{app:models:extraction}).

\paragraph{Start-End Prediction.}
\label{start_end}
Conceptually similar to the QA model described by \cite{devlin-etal-2019-bert}, we combine BERT with two final shared linear layers applied to all tokens. For each token, we predict if it is the start or the end of an attribute.

To account for the multiple possible answers, we use the sigmoid activation function and consider multiple start-end pairs.  We often observe very long predictions due to mismatched start and end tokens. This occurs when two attributes are predicted correctly, but the start of one attribute is matched with the end of the other, resulting in a very long token sequence. To combat this, we employ a heuristic post-processing method to form the final predictions (see Appendix \ref{postprocessing}).

\paragraph{Contiguous Prediction.}
To circumvent mismatching start and end tokens, we propose to consider each token separately. We append a linear layer after the BERT model, with which we independently classify whether each token is part of a given attribute. We then construct the predictions as all longest sequences of tokens that all have predicted probabilities greater than a threshold of $0.7$.
\begin{table}[b!]
    \begin{tabular}{lrrr}
        \toprule
        Method             & F1               & Prec.   & Rec.    \\
        \midrule
        TF-IDF + MLP       & $0.652$          & $0.768$ & $0.566$ \\
        BERT + Sigmoid & $0.672$          & $0.857$ & $0.553$ \\
        \hline
        $\text{BERT}_\text{SQ}$            & $0.838$          & $0.861$ & $0.816$ \\
        $\text{BERT}_\text{SQ}$ CL         & $0.844$          & $0.873$ & $0.816$ \\
        $\text{BERT}_\text{SQ}$ AD         & $0.857$          & $0.846$ & $0.868$ \\
        $\text{BERT}_\text{SQ}$ CL+AD      & $\mathbf{0.903}$ & $0.956$ & $0.855$ \\
        \bottomrule
    \end{tabular}
    \centering
    \caption{The F1 score, precision and recall of different methods on the Symptom Classification task. We denote the BERT Symptom Query model as $\text{BERT}_\text{SQ}$, the curriculum learning scheme as CL, and the augmented dataset as AD.}
    \label{symptom_classification_results}

\end{table}

\begin{table*}[t!]
\begin{tabular}{lrrrrr}
        \toprule
        Method            & Location            & Description       & Time           & Frequency      & Action        \\
        \midrule
        $\text{BERT}_\text{start\_end}$         & $0.56$            & $0.25$            & $\mathbf{0.42}$   &  $0.19 $          & $0.12$\\
        $\text{BERT}_\text{start\_end}$ general & $\mathbf{0.70}$   & $0.42$            & $0.29$            & $0.19$            & $0.27$\\
        $\text{BERT}_\text{contiguous}$         & $0.63$            & $\mathbf{0.43}$   & $0.31$            & $0.35$            & $0.21$\\
        $\text{BERT}_\text{contiguous}$ general & $0.66$            & $0.35$            & $0.34$            & $\mathbf{0.42}$   & $\mathbf{0.32}$  \\
        \bottomrule
    \end{tabular}
    \centering
    \caption{Attribute extraction results on the held out test set. We report the token-wise F1 score based on the constructed predictions. For precision and recall see Appendix \ref{app:more_results}.}
    \label{attribute_extraction_results}
\end{table*}
\FloatBarrier
\section{Experiments}

The dataset is split into train, test and validation sets. The test set is formed using $20\%$ of all labelings which were identified as correct after double labeling; we observe that these labelings are of higher quality. The validation set is formed using $10\%$ of all remaining labelings. We evaluate the model that achieved the highest validation F1 score (sum
of F1 scores over all attributes for the general attribute extraction models) and report it's scores on the held out test set. All F1 scores are micro averaged across all classes. All models are implemented using PyTorch (\citealt{paszke2019pytorch}).

\section{Results}
\paragraph{Symptom Classification.}
We observe that the symptom query model architecture outperforms both the TF-IDF + MLP and the sigmoid classifier baselines. Both curriculum learning and data augmentation appear to be beneficial and together yield a performance increase of $0.065$ F1 compared to the base BERT Symptom Query model (see Table \ref{symptom_classification_results}).

We further notice that the symptom query model with augmented descriptions generalizes better to unseen symptoms, as illustrated in Appendix \ref{app:generalization}.

\paragraph{Attribute Extraction.}
We report results by attribute type in Table \ref{attribute_extraction_results}. Both methods appear to produce useful predictions (see Appendix \ref{app:example_preds}). We observe that the ``general'' models that are trained on all attributes at once outperform the respective separately trained counterparts on most attributes.

The location attribute is predicted particularly well by almost all models, with F1 scores up to 0.7. We hypothesize that attributing a location to the correct symptom is simpler than correctly matching the other, more generic attributes: ``head'' is likely to be an attribute to ``headache'', while e.g. ``rarely'' could be a frequency attribute to almost any symptom.

\section{Conclusion}
In this work, we propose an approach to extract symptom information under the constraints of real-world utility and practicality. To the best of our knowledge, we are the first to attempt such a task in German. We demonstrate promising performance even on a relatively small dataset using a data augmentation technique and a curriculum learning scheme.

Although we observe limitations in model performance, particularly in the attribute extraction task, we are optimistic that this can be improved with a larger dataset. Further work could investigate the extraction of other relevant information such as pre-existing conditions and medication. With such improvements, we see potential for practical, automated anamnesis summarization in the near future.
\pagebreak
 \acks{
 We would like to thank Konstantinos Dimitriadis for the valuable domain insights and fruitful discussions.
 We are further grateful for Elliot Ash's support and the provided access to the Leonhard Cluster, as well as Julia Vogt's valuable feedback that improved the quality of the paper.
 }