text:
% Options for packages loaded elsewhere
\PassOptionsToPackage{unicode}{hyperref}
\PassOptionsToPackage{hyphens}{url}
%
\documentclass[
]{article}
\usepackage{lmodern}
\usepackage{amssymb,amsmath}
\usepackage{ifxetex,ifluatex}
\ifnum 0\ifxetex 1\fi\ifluatex 1\fi=0 % if pdftex
  \usepackage[T1]{fontenc}
  \usepackage[utf8]{inputenc}
  \usepackage{textcomp} % provide euro and other symbols
\else % if luatex or xetex
  \usepackage{unicode-math}
  \defaultfontfeatures{Scale=MatchLowercase}
  \defaultfontfeatures[\rmfamily]{Ligatures=TeX,Scale=1}
\fi
% Use upquote if available, for straight quotes in verbatim environments
\IfFileExists{upquote.sty}{\usepackage{upquote}}{}
\IfFileExists{microtype.sty}{% use microtype if available
  \usepackage[]{microtype}
  \UseMicrotypeSet[protrusion]{basicmath} % disable protrusion for tt fonts
}{}
\makeatletter
\@ifundefined{KOMAClassName}{% if non-KOMA class
  \IfFileExists{parskip.sty}{%
    \usepackage{parskip}
  }{% else
    \setlength{\parindent}{0pt}
    \setlength{\parskip}{6pt plus 2pt minus 1pt}}
}{% if KOMA class
  \KOMAoptions{parskip=half}}
\makeatother
\usepackage{xcolor}
\IfFileExists{xurl.sty}{\usepackage{xurl}}{} % add URL line breaks if available
\IfFileExists{bookmark.sty}{\usepackage{bookmark}}{\usepackage{hyperref}}
\hypersetup{
  pdftitle={Using ROhdsiWebApi},
  pdfauthor={Gowtham Rao},
  hidelinks,
  pdfcreator={LaTeX via pandoc}}
\urlstyle{same} % disable monospaced font for URLs
\usepackage[margin=1in]{geometry}
\usepackage{color}
\usepackage{fancyvrb}
\newcommand{\VerbBar}{|}
\newcommand{\VERB}{\Verb[commandchars=\\\{\}]}
\DefineVerbatimEnvironment{Highlighting}{Verbatim}{commandchars=\\\{\}}
% Add ',fontsize=\small' for more characters per line
\usepackage{framed}
\definecolor{shadecolor}{RGB}{248,248,248}
\newenvironment{Shaded}{\begin{snugshade}}{\end{snugshade}}
\newcommand{\AlertTok}[1]{\textcolor[rgb]{0.94,0.16,0.16}{#1}}
\newcommand{\AnnotationTok}[1]{\textcolor[rgb]{0.56,0.35,0.01}{\textbf{\textit{#1}}}}
\newcommand{\AttributeTok}[1]{\textcolor[rgb]{0.77,0.63,0.00}{#1}}
\newcommand{\BaseNTok}[1]{\textcolor[rgb]{0.00,0.00,0.81}{#1}}
\newcommand{\BuiltInTok}[1]{#1}
\newcommand{\CharTok}[1]{\textcolor[rgb]{0.31,0.60,0.02}{#1}}
\newcommand{\CommentTok}[1]{\textcolor[rgb]{0.56,0.35,0.01}{\textit{#1}}}
\newcommand{\CommentVarTok}[1]{\textcolor[rgb]{0.56,0.35,0.01}{\textbf{\textit{#1}}}}
\newcommand{\ConstantTok}[1]{\textcolor[rgb]{0.00,0.00,0.00}{#1}}
\newcommand{\ControlFlowTok}[1]{\textcolor[rgb]{0.13,0.29,0.53}{\textbf{#1}}}
\newcommand{\DataTypeTok}[1]{\textcolor[rgb]{0.13,0.29,0.53}{#1}}
\newcommand{\DecValTok}[1]{\textcolor[rgb]{0.00,0.00,0.81}{#1}}
\newcommand{\DocumentationTok}[1]{\textcolor[rgb]{0.56,0.35,0.01}{\textbf{\textit{#1}}}}
\newcommand{\ErrorTok}[1]{\textcolor[rgb]{0.64,0.00,0.00}{\textbf{#1}}}
\newcommand{\ExtensionTok}[1]{#1}
\newcommand{\FloatTok}[1]{\textcolor[rgb]{0.00,0.00,0.81}{#1}}
\newcommand{\FunctionTok}[1]{\textcolor[rgb]{0.00,0.00,0.00}{#1}}
\newcommand{\ImportTok}[1]{#1}
\newcommand{\InformationTok}[1]{\textcolor[rgb]{0.56,0.35,0.01}{\textbf{\textit{#1}}}}
\newcommand{\KeywordTok}[1]{\textcolor[rgb]{0.13,0.29,0.53}{\textbf{#1}}}
\newcommand{\NormalTok}[1]{#1}
\newcommand{\OperatorTok}[1]{\textcolor[rgb]{0.81,0.36,0.00}{\textbf{#1}}}
\newcommand{\OtherTok}[1]{\textcolor[rgb]{0.56,0.35,0.01}{#1}}
\newcommand{\PreprocessorTok}[1]{\textcolor[rgb]{0.56,0.35,0.01}{\textit{#1}}}
\newcommand{\RegionMarkerTok}[1]{#1}
\newcommand{\SpecialCharTok}[1]{\textcolor[rgb]{0.00,0.00,0.00}{#1}}
\newcommand{\SpecialStringTok}[1]{\textcolor[rgb]{0.31,0.60,0.02}{#1}}
\newcommand{\StringTok}[1]{\textcolor[rgb]{0.31,0.60,0.02}{#1}}
\newcommand{\VariableTok}[1]{\textcolor[rgb]{0.00,0.00,0.00}{#1}}
\newcommand{\VerbatimStringTok}[1]{\textcolor[rgb]{0.31,0.60,0.02}{#1}}
\newcommand{\WarningTok}[1]{\textcolor[rgb]{0.56,0.35,0.01}{\textbf{\textit{#1}}}}
\usepackage{longtable,booktabs}
% Correct order of tables after \paragraph or \subparagraph
\usepackage{etoolbox}
\makeatletter
\patchcmd\longtable{\par}{\if@noskipsec\mbox{}\fi\par}{}{}
\makeatother
% Allow footnotes in longtable head/foot
\IfFileExists{footnotehyper.sty}{\usepackage{footnotehyper}}{\usepackage{footnote}}
\makesavenoteenv{longtable}
\usepackage{graphicx,grffile}
\makeatletter
\def\maxwidth{\ifdim\Gin@nat@width>\linewidth\linewidth\else\Gin@nat@width\fi}
\def\maxheight{\ifdim\Gin@nat@height>\textheight\textheight\else\Gin@nat@height\fi}
\makeatother
% Scale images if necessary, so that they will not overflow the page
% margins by default, and it is still possible to overwrite the defaults
% using explicit options in \includegraphics[width, height, ...]{}
\setkeys{Gin}{width=\maxwidth,height=\maxheight,keepaspectratio}
% Set default figure placement to htbp
\makeatletter
\def\fps@figure{htbp}
\makeatother
\setlength{\emergencystretch}{3em} % prevent overfull lines
\providecommand{\tightlist}{%
  \setlength{\itemsep}{0pt}\setlength{\parskip}{0pt}}
\setcounter{secnumdepth}{5}

\title{Using ROhdsiWebApi}
\author{Gowtham Rao}
\date{2020-11-25}

\begin{document}
\maketitle

{
\setcounter{tocdepth}{2}
\tableofcontents
}
ROhdsiWebApi is part of \href{https://ohdsi.github.io/Hades}{HADES}.

\hypertarget{introduction}{%
\section{Introduction}\label{introduction}}

From Package
\href{https://github.com/OHDSI/ROhdsiWebApi/blob/master/README.md}{Readme}

\begin{quote}
ROhdsiWebApi is a R based interface to
\href{https://github.com/ohdsi/webapi}{`WebApi'} (OHDSI RESTful
services), and performs GET/PULL/POST/DELETE calls via the WebApi. All
objects starting from R or output to R - are analysis ready R-objects
like list and data.frame. The package handles the intermediary steps by
converting R-objects to JSON and vice versa. To ensure r-objects are
analysis ready, the objects are type converted where possible,
e.g.~date/date time are converted from string to POSIXct.
\end{quote}

\begin{quote}
This package makes reproducible research easier, by offering ability to
retrieve detailed study specifications, transport study specifications
from one instance to another, programmatically invoke the generation of
a sequence of steps that are part of a study, manage running studies in
batch mode.
\end{quote}

This document will attempt to explain how ROhdsiWebApi maybe used to
achieve reproducible research.

\hypertarget{webapi-configurations-and-rohdsiwebapi}{%
\section{WebApi configurations and
ROhdsiWebApi}\label{webapi-configurations-and-rohdsiwebapi}}

To successfully use ROhdsiWebApi, it is necessary to have an active
\href{https://github.com/ohdsi/webapi}{`WebApi'} endpoint with a known
baseUrl such as ``\url{http://server.org:80/WebAPI}''.
\href{https://github.com/ohdsi/webapi}{`WebApi'} has many functional
categories.

To ensure reproducibility of work it is best to know the version of the
WebApi (i.e.~Atlas backend) being used. An easy way to do that is (and
output maybe included in your study results)

\begin{Shaded}
\begin{Highlighting}[]

\NormalTok{version <-}\StringTok{ }\NormalTok{ROhdsiWebApi}\OperatorTok{:::}\KeywordTok{getWebApiVersion}\NormalTok{(}\DataTypeTok{baseUrl =}\NormalTok{ baseUrl)}
\NormalTok{message1 <-}\StringTok{ }\KeywordTok{paste0}\NormalTok{(}\StringTok{'This Vignette was created using WebApi version: '}\NormalTok{, }
\NormalTok{                   version, }
                   \StringTok{' on baseUrl: '}\NormalTok{, }
\NormalTok{                   baseUrl, }
                   \StringTok{". The CDM had the following source data configured: "}\NormalTok{)}
\NormalTok{cdmSources <-}\StringTok{ }\NormalTok{ROhdsiWebApi}\OperatorTok{::}\KeywordTok{getCdmSources}\NormalTok{(}\DataTypeTok{baseUrl =}\NormalTok{ baseUrl)}
\NormalTok{priorityVocabulary <-}\StringTok{ }\NormalTok{ROhdsiWebApi}\OperatorTok{::}\KeywordTok{getPriorityVocabularyKey}\NormalTok{(}\DataTypeTok{baseUrl =}\NormalTok{ baseUrl)}
\end{Highlighting}
\end{Shaded}

The object \texttt{version} will show your webApi version. Example: This
Vignette was created using WebApi version: 2.8.0 on baseUrl:
\url{https://epi.jnj.com:8443/WebAPI}. The CDM had the following source
data configured: .

\begin{Shaded}
\begin{Highlighting}[]

\NormalTok{cdmSources}
\CommentTok{#> # A tibble: 58 x 7}
\CommentTok{#>    sourceId sourceName          sourceKey           sourceDialect cdmDatabaseSchema        vocabDatabaseSchema     resultsDatabaseSchema       }
\CommentTok{#>       <int> <chr>               <chr>               <chr>         <chr>                    <chr>                   <chr>                       }
\CommentTok{#>  1      254 CPRD ACE ARB (v121~ CDM_CPRD_ACE_ARB_v~ pdw           CDM_CPRD_ACE_ARB_v1213.~ CDM_CPRD_ACE_ARB_v1213~ CDM_CPRD_ACE_ARB_v1213.ohds~}
\CommentTok{#>  2      228 CPRD (v1102)        CDM_CPRD_V1102      pdw           CDM_CPRD_V1102.dbo       CDM_CPRD_V1102.dbo      CDM_CPRD_V1102.ohdsi_results}
\CommentTok{#>  3      237 CPRD (v1150)        CDM_CPRD_V1150      pdw           CDM_CPRD_V1150.dbo       CDM_CPRD_V1150.dbo      CDM_CPRD_V1150.ohdsi_results}
\CommentTok{#>  4      172 HCUP (v869)         CDM_HCUP_V869       pdw           CDM_HCUP_V869.dbo        CDM_HCUP_V869.dbo       CDM_HCUP_V869.ohdsi_results }
\CommentTok{#>  5      278 HEALTH VERITY COVI~ CDM_HEALTHVERITY_C~ pdw           CDM_HealthVerity_COVID_~ CDM_HealthVerity_COVID~ CDM_HealthVerity_COVID_V130~}
\CommentTok{#>  6      238 IBM CCAE (v1151)    CDM_IBM_CCAE_V1151  pdw           CDM_IBM_CCAE_V1151.dbo   CDM_IBM_CCAE_V1151.dbo  CDM_IBM_CCAE_V1151.ohdsi_re~}
\CommentTok{#>  7      248 IBM CCAE (v1191)    CDM_IBM_CCAE_V1191  pdw           CDM_IBM_CCAE_V1191.dbo   CDM_IBM_CCAE_V1191.dbo  CDM_IBM_CCAE_V1191.ohdsi_re~}
\CommentTok{#>  8      271 IBM CCAE (v1247)    CDM_IBM_CCAE_V1247  pdw           CDM_IBM_CCAE_V1247.dbo   CDM_IBM_CCAE_V1247.dbo  CDM_IBM_CCAE_V1247.ohdsi_re~}
\CommentTok{#>  9      231 IBM MDCD (v1105)    CDM_IBM_MDCD_V1105  pdw           CDM_IBM_MDCD_V1105.dbo   CDM_IBM_MDCD_V1105.dbo  CDM_IBM_MDCD_V1105.ohdsi_re~}
\CommentTok{#> 10      240 IBM MDCD (v1153)    CDM_IBM_MDCD_V1153  pdw           CDM_IBM_MDCD_V1153.dbo   CDM_IBM_MDCD_V1153.dbo  CDM_IBM_MDCD_V1153.ohdsi_re~}
\CommentTok{#> # ... with 48 more rows}
\end{Highlighting}
\end{Shaded}

The priority vocabulary for the WebApi is VOCABULARY\_20200525.

We can also perform checks on the WebApi, example - we may want to see
if the `HCUP' \& `SYNPUF1K' is a valid SourceKey in the current webApi.

\begin{Shaded}
\begin{Highlighting}[]

\NormalTok{ROhdsiWebApi}\OperatorTok{::}\KeywordTok{isValidSourceKey}\NormalTok{(}\DataTypeTok{sourceKeys =} \KeywordTok{c}\NormalTok{(}\StringTok{'HCUP'}\NormalTok{, }\StringTok{'SYNPUF1K'}\NormalTok{), }\DataTypeTok{baseUrl =}\NormalTok{ baseUrl)}
\CommentTok{#> [1]  TRUE FALSE}
\end{Highlighting}
\end{Shaded}

\hypertarget{webapi-analytical-categories.}{%
\subsection{WebApi Analytical
categories.}\label{webapi-analytical-categories.}}

WebApi maybe considered to have certain modular analytic categories.
ROhdsiWebApi supports the following categories:

\begin{longtable}[]{@{}ll@{}}
\toprule
\begin{minipage}[b]{0.21\columnwidth}\raggedright
Category\strut
\end{minipage} & \begin{minipage}[b]{0.73\columnwidth}\raggedright
Features\strut
\end{minipage}\tabularnewline
\midrule
\endhead
\begin{minipage}[t]{0.21\columnwidth}\raggedright
ConceptSet\strut
\end{minipage} & \begin{minipage}[t]{0.73\columnwidth}\raggedright
Functions for interfacing with ConceptSet in WebApi\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.21\columnwidth}\raggedright
Cohort\strut
\end{minipage} & \begin{minipage}[t]{0.73\columnwidth}\raggedright
Functions for interfacing with Cohort in WebApi\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.21\columnwidth}\raggedright
IncidenceRate\strut
\end{minipage} & \begin{minipage}[t]{0.73\columnwidth}\raggedright
Functions for interfacing with IncidenceRate in WebApi\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.21\columnwidth}\raggedright
Estimation\strut
\end{minipage} & \begin{minipage}[t]{0.73\columnwidth}\raggedright
Functions for interfacing with Estimation in WebApi\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.21\columnwidth}\raggedright
Prediction\strut
\end{minipage} & \begin{minipage}[t]{0.73\columnwidth}\raggedright
Functions for interfacing with Prediction in WebApi\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.21\columnwidth}\raggedright
Characterization\strut
\end{minipage} & \begin{minipage}[t]{0.73\columnwidth}\raggedright
Functions for interfacing with Characterization in WebApi\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.21\columnwidth}\raggedright
Pathway\strut
\end{minipage} & \begin{minipage}[t]{0.73\columnwidth}\raggedright
Functions for interfacing with Pathway in WebApi\strut
\end{minipage}\tabularnewline
\bottomrule
\end{longtable}

\hypertarget{framework-of-rohdsiwebapi}{%
\section{Framework of ROhdsiWebApi}\label{framework-of-rohdsiwebapi}}

ROhdsiWebApi maybe better understood by having atleast a high level
understanding of
\href{https://en.wikipedia.org/wiki/Create,_read,_update_and_delete}{CRUD}
framework for \href{https://github.com/ohdsi/webapi}{WebApi}, i.e.~the
GET, PUT, DELETE, POST calls to the API. See
\href{http://webapidoc.ohdsi.org/index.html}{the documentation of the
WebApi}.

For each supported category, ROhdsiWebAPi performs GET, PUT, DELETE,
POST calls to WebApi in background. The details of what calls are
actually performed is less important to an analyst, but it is useful to
understand the naming conventions of ROhdsiWebApi.

\hypertarget{naming-conventions-of-rohdsiwebapi}{%
\subsection{Naming conventions of
ROhdsiWebApi}\label{naming-conventions-of-rohdsiwebapi}}

Most functions in ROhdsiWebApi start with an action oriented `verb' -
such as

\begin{longtable}[]{@{}ll@{}}
\toprule
\begin{minipage}[b]{0.46\columnwidth}\raggedright
Function Name\strut
\end{minipage} & \begin{minipage}[b]{0.48\columnwidth}\raggedright
Description\strut
\end{minipage}\tabularnewline
\midrule
\endhead
\begin{minipage}[t]{0.46\columnwidth}\raggedright
cancelCharacterizationGeneration\strut
\end{minipage} & \begin{minipage}[t]{0.48\columnwidth}\raggedright
Cancel Characterization Generation\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.46\columnwidth}\raggedright
cancelCohortGeneration\strut
\end{minipage} & \begin{minipage}[t]{0.48\columnwidth}\raggedright
Cancel Cohort Generation\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.46\columnwidth}\raggedright
cancelGeneration\strut
\end{minipage} & \begin{minipage}[t]{0.48\columnwidth}\raggedright
Cancel Generation\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.46\columnwidth}\raggedright
cancelIncidenceRateGeneration\strut
\end{minipage} & \begin{minipage}[t]{0.48\columnwidth}\raggedright
Cancel Incidence Rate Generation\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.46\columnwidth}\raggedright
cancelPathwayGeneration\strut
\end{minipage} & \begin{minipage}[t]{0.48\columnwidth}\raggedright
Cancel Pathway Generation\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.46\columnwidth}\raggedright
checkInputFileEncoding\strut
\end{minipage} & \begin{minipage}[t]{0.48\columnwidth}\raggedright
Check Input File Encoding\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.46\columnwidth}\raggedright
convertConceptSetDefinitionToTable\strut
\end{minipage} & \begin{minipage}[t]{0.48\columnwidth}\raggedright
Convert Concept Set Definition To Table\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.46\columnwidth}\raggedright
createConceptSetWorkbook\strut
\end{minipage} & \begin{minipage}[t]{0.48\columnwidth}\raggedright
Create Concept Set Workbook\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.46\columnwidth}\raggedright
deleteCharacterizationDefinition\strut
\end{minipage} & \begin{minipage}[t]{0.48\columnwidth}\raggedright
Delete Characterization Definition\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.46\columnwidth}\raggedright
deleteCohortDefinition\strut
\end{minipage} & \begin{minipage}[t]{0.48\columnwidth}\raggedright
Delete Cohort Definition\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.46\columnwidth}\raggedright
deleteConceptSetDefinition\strut
\end{minipage} & \begin{minipage}[t]{0.48\columnwidth}\raggedright
Delete Concept Set Definition\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.46\columnwidth}\raggedright
deleteDefinition\strut
\end{minipage} & \begin{minipage}[t]{0.48\columnwidth}\raggedright
Delete Definition\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.46\columnwidth}\raggedright
deleteEstimationDefinition\strut
\end{minipage} & \begin{minipage}[t]{0.48\columnwidth}\raggedright
Delete Estimation Definition\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.46\columnwidth}\raggedright
deleteIncidenceRateDefinition\strut
\end{minipage} & \begin{minipage}[t]{0.48\columnwidth}\raggedright
Delete Incidence Rate Definition\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.46\columnwidth}\raggedright
deletePathwayDefinition\strut
\end{minipage} & \begin{minipage}[t]{0.48\columnwidth}\raggedright
Delete Pathway Definition\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.46\columnwidth}\raggedright
deletePredictionDefinition\strut
\end{minipage} & \begin{minipage}[t]{0.48\columnwidth}\raggedright
Delete Prediction Definition\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.46\columnwidth}\raggedright
detectCharacterizationsByName\strut
\end{minipage} & \begin{minipage}[t]{0.48\columnwidth}\raggedright
Detect Characterizations By Name\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.46\columnwidth}\raggedright
detectCohortsByName\strut
\end{minipage} & \begin{minipage}[t]{0.48\columnwidth}\raggedright
Detect Cohorts By Name\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.46\columnwidth}\raggedright
detectConceptSetsByName\strut
\end{minipage} & \begin{minipage}[t]{0.48\columnwidth}\raggedright
Detect Concept Sets By Name\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.46\columnwidth}\raggedright
detectEstimationsByName\strut
\end{minipage} & \begin{minipage}[t]{0.48\columnwidth}\raggedright
Detect Estimations By Name\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.46\columnwidth}\raggedright
detectIncidenceRatesByName\strut
\end{minipage} & \begin{minipage}[t]{0.48\columnwidth}\raggedright
Detect Incidence Rates By Name\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.46\columnwidth}\raggedright
detectPathwaysByName\strut
\end{minipage} & \begin{minipage}[t]{0.48\columnwidth}\raggedright
Detect Pathways By Name\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.46\columnwidth}\raggedright
detectPredictionsByName\strut
\end{minipage} & \begin{minipage}[t]{0.48\columnwidth}\raggedright
Detect Predictions By Name\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.46\columnwidth}\raggedright
existsCharacterizationName\strut
\end{minipage} & \begin{minipage}[t]{0.48\columnwidth}\raggedright
Exists Characterization Name\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.46\columnwidth}\raggedright
existsCohortName\strut
\end{minipage} & \begin{minipage}[t]{0.48\columnwidth}\raggedright
Exists Cohort Name\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.46\columnwidth}\raggedright
existsConceptSetName\strut
\end{minipage} & \begin{minipage}[t]{0.48\columnwidth}\raggedright
Exists Concept Set Name\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.46\columnwidth}\raggedright
existsEstimationName\strut
\end{minipage} & \begin{minipage}[t]{0.48\columnwidth}\raggedright
Exists Estimation Name\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.46\columnwidth}\raggedright
existsIncidenceRateName\strut
\end{minipage} & \begin{minipage}[t]{0.48\columnwidth}\raggedright
Exists Incidence Rate Name\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.46\columnwidth}\raggedright
existsPathwayName\strut
\end{minipage} & \begin{minipage}[t]{0.48\columnwidth}\raggedright
Exists Pathway Name\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.46\columnwidth}\raggedright
existsPredictionName\strut
\end{minipage} & \begin{minipage}[t]{0.48\columnwidth}\raggedright
Exists Prediction Name\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.46\columnwidth}\raggedright
getCdmsources\strut
\end{minipage} & \begin{minipage}[t]{0.48\columnwidth}\raggedright
Get Cdmsources\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.46\columnwidth}\raggedright
getCharacterizationDefinition\strut
\end{minipage} & \begin{minipage}[t]{0.48\columnwidth}\raggedright
Get Characterization Definition\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.46\columnwidth}\raggedright
getCharacterizationDefinitionsMetadata\strut
\end{minipage} & \begin{minipage}[t]{0.48\columnwidth}\raggedright
Get Characterization Definitions Metadata\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.46\columnwidth}\raggedright
getCharacterizationGenerationinformation\strut
\end{minipage} & \begin{minipage}[t]{0.48\columnwidth}\raggedright
Get Characterization Generationinformation\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.46\columnwidth}\raggedright
getCharacterizationResults\strut
\end{minipage} & \begin{minipage}[t]{0.48\columnwidth}\raggedright
Get Characterization Results\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.46\columnwidth}\raggedright
getCohortDefinition\strut
\end{minipage} & \begin{minipage}[t]{0.48\columnwidth}\raggedright
Get Cohort Definition\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.46\columnwidth}\raggedright
getCohortDefinitionExpression\strut
\end{minipage} & \begin{minipage}[t]{0.48\columnwidth}\raggedright
Get Cohort Definition Expression\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.46\columnwidth}\raggedright
getCohortDefinitionName\strut
\end{minipage} & \begin{minipage}[t]{0.48\columnwidth}\raggedright
Get Cohort Definition Name\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.46\columnwidth}\raggedright
getCohortDefinitionSql\strut
\end{minipage} & \begin{minipage}[t]{0.48\columnwidth}\raggedright
Get Cohort Definition Sql\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.46\columnwidth}\raggedright
getCohortDefinitionsMetadata\strut
\end{minipage} & \begin{minipage}[t]{0.48\columnwidth}\raggedright
Get Cohort Definitions Metadata\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.46\columnwidth}\raggedright
getCohortGenerationinformation\strut
\end{minipage} & \begin{minipage}[t]{0.48\columnwidth}\raggedright
Get Cohort Generationinformation\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.46\columnwidth}\raggedright
getCohortInclusionrulesandcounts\strut
\end{minipage} & \begin{minipage}[t]{0.48\columnwidth}\raggedright
Get Cohort Inclusionrulesandcounts\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.46\columnwidth}\raggedright
getCohortResults\strut
\end{minipage} & \begin{minipage}[t]{0.48\columnwidth}\raggedright
Get Cohort Results\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.46\columnwidth}\raggedright
getCohortSql\strut
\end{minipage} & \begin{minipage}[t]{0.48\columnwidth}\raggedright
Get Cohort Sql\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.46\columnwidth}\raggedright
getConceptSetDefinition\strut
\end{minipage} & \begin{minipage}[t]{0.48\columnwidth}\raggedright
Get Concept Set Definition\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.46\columnwidth}\raggedright
getConceptSetDefinitionsMetadata\strut
\end{minipage} & \begin{minipage}[t]{0.48\columnwidth}\raggedright
Get Concept Set Definitions Metadata\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.46\columnwidth}\raggedright
getConceptSetsFromCohortDefinition\strut
\end{minipage} & \begin{minipage}[t]{0.48\columnwidth}\raggedright
Get Concept Sets From Cohort Definition\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.46\columnwidth}\raggedright
getConcepts\strut
\end{minipage} & \begin{minipage}[t]{0.48\columnwidth}\raggedright
Get Concepts\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.46\columnwidth}\raggedright
getDefinition\strut
\end{minipage} & \begin{minipage}[t]{0.48\columnwidth}\raggedright
Get Definition\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.46\columnwidth}\raggedright
getDefinitionsMetadata\strut
\end{minipage} & \begin{minipage}[t]{0.48\columnwidth}\raggedright
Get Definitions Metadata\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.46\columnwidth}\raggedright
getEstimationDefinition\strut
\end{minipage} & \begin{minipage}[t]{0.48\columnwidth}\raggedright
Get Estimation Definition\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.46\columnwidth}\raggedright
getEstimationDefinitionsMetadata\strut
\end{minipage} & \begin{minipage}[t]{0.48\columnwidth}\raggedright
Get Estimation Definitions Metadata\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.46\columnwidth}\raggedright
getGenerationinformation\strut
\end{minipage} & \begin{minipage}[t]{0.48\columnwidth}\raggedright
Get Generationinformation\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.46\columnwidth}\raggedright
getIncidenceRateDefinition\strut
\end{minipage} & \begin{minipage}[t]{0.48\columnwidth}\raggedright
Get Incidence Rate Definition\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.46\columnwidth}\raggedright
getIncidenceRateDefinitionsMetadata\strut
\end{minipage} & \begin{minipage}[t]{0.48\columnwidth}\raggedright
Get Incidence Rate Definitions Metadata\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.46\columnwidth}\raggedright
getIncidenceRateGenerationinformation\strut
\end{minipage} & \begin{minipage}[t]{0.48\columnwidth}\raggedright
Get Incidence Rate Generationinformation\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.46\columnwidth}\raggedright
getIncidenceRateResults\strut
\end{minipage} & \begin{minipage}[t]{0.48\columnwidth}\raggedright
Get Incidence Rate Results\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.46\columnwidth}\raggedright
getPathwayDefinition\strut
\end{minipage} & \begin{minipage}[t]{0.48\columnwidth}\raggedright
Get Pathway Definition\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.46\columnwidth}\raggedright
getPathwayDefinitionsMetadata\strut
\end{minipage} & \begin{minipage}[t]{0.48\columnwidth}\raggedright
Get Pathway Definitions Metadata\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.46\columnwidth}\raggedright
getPathwayGenerationinformation\strut
\end{minipage} & \begin{minipage}[t]{0.48\columnwidth}\raggedright
Get Pathway Generationinformation\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.46\columnwidth}\raggedright
getPathwayResults\strut
\end{minipage} & \begin{minipage}[t]{0.48\columnwidth}\raggedright
Get Pathway Results\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.46\columnwidth}\raggedright
getPersonProfile\strut
\end{minipage} & \begin{minipage}[t]{0.48\columnwidth}\raggedright
Get Person Profile\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.46\columnwidth}\raggedright
getPredictionDefinition\strut
\end{minipage} & \begin{minipage}[t]{0.48\columnwidth}\raggedright
Get Prediction Definition\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.46\columnwidth}\raggedright
getPredictionDefinitionsMetadata\strut
\end{minipage} & \begin{minipage}[t]{0.48\columnwidth}\raggedright
Get Prediction Definitions Metadata\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.46\columnwidth}\raggedright
getPriorityvocabularykey\strut
\end{minipage} & \begin{minipage}[t]{0.48\columnwidth}\raggedright
Get Priorityvocabularykey\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.46\columnwidth}\raggedright
getResults\strut
\end{minipage} & \begin{minipage}[t]{0.48\columnwidth}\raggedright
Get Results\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.46\columnwidth}\raggedright
getSourceconcepts\strut
\end{minipage} & \begin{minipage}[t]{0.48\columnwidth}\raggedright
Get Sourceconcepts\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.46\columnwidth}\raggedright
getWebApiVersion\strut
\end{minipage} & \begin{minipage}[t]{0.48\columnwidth}\raggedright
Get Web Api Version\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.46\columnwidth}\raggedright
insertCohortDefinitionInPackage\strut
\end{minipage} & \begin{minipage}[t]{0.48\columnwidth}\raggedright
Insert Cohort Definition In Package\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.46\columnwidth}\raggedright
insertCohortDefinitionSetInPackage\strut
\end{minipage} & \begin{minipage}[t]{0.48\columnwidth}\raggedright
Insert Cohort Definition Set In Package\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.46\columnwidth}\raggedright
invokeCharacterizationGeneration\strut
\end{minipage} & \begin{minipage}[t]{0.48\columnwidth}\raggedright
Invoke Characterization Generation\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.46\columnwidth}\raggedright
invokeCohortGeneration\strut
\end{minipage} & \begin{minipage}[t]{0.48\columnwidth}\raggedright
Invoke Cohort Generation\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.46\columnwidth}\raggedright
invokeGeneration\strut
\end{minipage} & \begin{minipage}[t]{0.48\columnwidth}\raggedright
Invoke Generation\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.46\columnwidth}\raggedright
invokeIncidenceRateGeneration\strut
\end{minipage} & \begin{minipage}[t]{0.48\columnwidth}\raggedright
Invoke Incidence Rate Generation\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.46\columnwidth}\raggedright
invokePathwayGeneration\strut
\end{minipage} & \begin{minipage}[t]{0.48\columnwidth}\raggedright
Invoke Pathway Generation\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.46\columnwidth}\raggedright
isvalidCharacterizationId\strut
\end{minipage} & \begin{minipage}[t]{0.48\columnwidth}\raggedright
Isvalid Characterization Id\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.46\columnwidth}\raggedright
isvalidCohortId\strut
\end{minipage} & \begin{minipage}[t]{0.48\columnwidth}\raggedright
Isvalid Cohort Id\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.46\columnwidth}\raggedright
isvalidConceptSetId\strut
\end{minipage} & \begin{minipage}[t]{0.48\columnwidth}\raggedright
Isvalid Concept Set Id\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.46\columnwidth}\raggedright
isvalidEstimationId\strut
\end{minipage} & \begin{minipage}[t]{0.48\columnwidth}\raggedright
Isvalid Estimation Id\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.46\columnwidth}\raggedright
isvalidId\strut
\end{minipage} & \begin{minipage}[t]{0.48\columnwidth}\raggedright
Isvalid Id\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.46\columnwidth}\raggedright
isvalidIncidenceRateId\strut
\end{minipage} & \begin{minipage}[t]{0.48\columnwidth}\raggedright
Isvalid Incidence Rate Id\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.46\columnwidth}\raggedright
isvalidPathwayId\strut
\end{minipage} & \begin{minipage}[t]{0.48\columnwidth}\raggedright
Isvalid Pathway Id\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.46\columnwidth}\raggedright
isvalidPredictionId\strut
\end{minipage} & \begin{minipage}[t]{0.48\columnwidth}\raggedright
Isvalid Prediction Id\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.46\columnwidth}\raggedright
isvalidSourceKey\strut
\end{minipage} & \begin{minipage}[t]{0.48\columnwidth}\raggedright
Isvalid Source Key\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.46\columnwidth}\raggedright
postCharacterizationDefinition\strut
\end{minipage} & \begin{minipage}[t]{0.48\columnwidth}\raggedright
Post Characterization Definition\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.46\columnwidth}\raggedright
postCohortDefinition\strut
\end{minipage} & \begin{minipage}[t]{0.48\columnwidth}\raggedright
Post Cohort Definition\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.46\columnwidth}\raggedright
postConceptSetDefinition\strut
\end{minipage} & \begin{minipage}[t]{0.48\columnwidth}\raggedright
Post Concept Set Definition\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.46\columnwidth}\raggedright
postDefinition\strut
\end{minipage} & \begin{minipage}[t]{0.48\columnwidth}\raggedright
Post Definition\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.46\columnwidth}\raggedright
postEstimationDefinition\strut
\end{minipage} & \begin{minipage}[t]{0.48\columnwidth}\raggedright
Post Estimation Definition\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.46\columnwidth}\raggedright
postIncidenceRateDefinition\strut
\end{minipage} & \begin{minipage}[t]{0.48\columnwidth}\raggedright
Post Incidence Rate Definition\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.46\columnwidth}\raggedright
postPathwayDefinition\strut
\end{minipage} & \begin{minipage}[t]{0.48\columnwidth}\raggedright
Post Pathway Definition\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.46\columnwidth}\raggedright
postPredictionDefinition\strut
\end{minipage} & \begin{minipage}[t]{0.48\columnwidth}\raggedright
Post Prediction Definition\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.46\columnwidth}\raggedright
resolveConceptSet\strut
\end{minipage} & \begin{minipage}[t]{0.48\columnwidth}\raggedright
Resolve Concept Set\strut
\end{minipage}\tabularnewline
\bottomrule
\end{longtable}

Most of the functions start with the following verbs:

\begin{longtable}[]{@{}lr@{}}
\toprule
Function Verb & Number Of Functions\tabularnewline
\midrule
\endhead
Get & 38\tabularnewline
Isvalid & 9\tabularnewline
Delete & 8\tabularnewline
Post & 8\tabularnewline
Detect & 7\tabularnewline
Exists & 7\tabularnewline
Cancel & 5\tabularnewline
Invoke & 5\tabularnewline
Insert & 2\tabularnewline
Check & 1\tabularnewline
Convert & 1\tabularnewline
Create & 1\tabularnewline
Resolve & 1\tabularnewline
\bottomrule
\end{longtable}

A function to get \texttt{Definition} is \texttt{getDefinitionMetaData}
function. This is a general function that is able to get the Metadata
for all specifications within a category.

\begin{Shaded}
\begin{Highlighting}[]

\NormalTok{ROhdsiWebApi}\OperatorTok{::}\KeywordTok{getDefinitionsMetadata}\NormalTok{(}\DataTypeTok{baseUrl =}\NormalTok{ baseUrl, }
                                     \DataTypeTok{category =} \StringTok{'cohort'}\NormalTok{) }\OperatorTok{%>%}
\StringTok{  }\KeywordTok{arrange}\NormalTok{(.data}\OperatorTok{$}\NormalTok{id) }\OperatorTok{%>%}
\StringTok{  }\KeywordTok{rename_all}\NormalTok{(}\DataTypeTok{.funs =}\NormalTok{ SqlRender}\OperatorTok{::}\NormalTok{camelCaseToTitleCase) }\OperatorTok{%>%}
\StringTok{  }\KeywordTok{tail}\NormalTok{() }
\CommentTok{#> # A tibble: 6 x 6}
\CommentTok{#>      Id Name                Description                                                `Created Date`      `Modified Date`     `Has Write Acce~}
\CommentTok{#>   <int> <chr>               <chr>                                                      <dttm>              <dttm>              <lgl>           }
\CommentTok{#> 1 19665 [AM v1] Asthma tre~ "Patients with newly diagnosed asthma (with all descendan~ 2020-11-25 11:14:00 2020-11-25 13:42:00 FALSE           }
\CommentTok{#> 2 19666 [AM v1] COPD treat~ "Patients with newly diagnosed asthma (with all descendan~ 2020-11-25 11:14:00 NA                  FALSE           }
\CommentTok{#> 3 19667 [AM v1] Asthma COP~ "Patients with newly diagnosed asthma (with all descendan~ 2020-11-25 11:14:00 2020-11-25 12:17:00 FALSE           }
\CommentTok{#> 4 19669 [460] Atrial Fibri~ "[460] Atrial Fibrillation xSpec Model"                    2020-11-25 11:59:00 2020-11-25 12:54:00 FALSE           }
\CommentTok{#> 5 19673 [Epi 520] Major Bl~ "Major bleeding events using the Cunningham definition. T~ 2020-11-25 14:00:00 2020-11-25 14:08:00 FALSE           }
\CommentTok{#> 6 19674 [460] Rituximab Pl~  <NA>                                                      2020-11-25 16:08:00 NA                  FALSE}
\end{Highlighting}
\end{Shaded}

The same output may be achieved using

\begin{Shaded}
\begin{Highlighting}[]

\NormalTok{ROhdsiWebApi}\OperatorTok{::}\KeywordTok{getCohortDefinitionsMetaData}\NormalTok{(}\DataTypeTok{baseUrl =}\NormalTok{ baseUrl) }\OperatorTok{%>%}\StringTok{ }
\StringTok{  }\KeywordTok{arrange}\NormalTok{(.data}\OperatorTok{$}\NormalTok{id) }\OperatorTok{%>%}\StringTok{ }
\StringTok{  }\KeywordTok{rename_all}\NormalTok{(}\DataTypeTok{.funs =}\NormalTok{ SqlRender}\OperatorTok{::}\NormalTok{camelCaseToTitleCase) }\OperatorTok{%>%}\StringTok{ }
\StringTok{  }\KeywordTok{tail}\NormalTok{() }
\CommentTok{#> # A tibble: 6 x 6}
\CommentTok{#>      Id Name                Description                                                `Created Date`      `Modified Date`     `Has Write Acce~}
\CommentTok{#>   <int> <chr>               <chr>                                                      <dttm>              <dttm>              <lgl>           }
\CommentTok{#> 1 19665 [AM v1] Asthma tre~ "Patients with newly diagnosed asthma (with all descendan~ 2020-11-25 11:14:00 2020-11-25 13:42:00 FALSE           }
\CommentTok{#> 2 19666 [AM v1] COPD treat~ "Patients with newly diagnosed asthma (with all descendan~ 2020-11-25 11:14:00 NA                  FALSE           }
\CommentTok{#> 3 19667 [AM v1] Asthma COP~ "Patients with newly diagnosed asthma (with all descendan~ 2020-11-25 11:14:00 2020-11-25 12:17:00 FALSE           }
\CommentTok{#> 4 19669 [460] Atrial Fibri~ "[460] Atrial Fibrillation xSpec Model"                    2020-11-25 11:59:00 2020-11-25 12:54:00 FALSE           }
\CommentTok{#> 5 19673 [Epi 520] Major Bl~ "Major bleeding events using the Cunningham definition. T~ 2020-11-25 14:00:00 2020-11-25 14:08:00 FALSE           }
\CommentTok{#> 6 19674 [460] Rituximab Pl~  <NA>                                                      2020-11-25 16:08:00 NA                  FALSE}
\end{Highlighting}
\end{Shaded}

Similar approach may be used for all categories as follows:

\begin{Shaded}
\begin{Highlighting}[]

\NormalTok{ROhdsiWebApi}\OperatorTok{::}\KeywordTok{getDefinitionsMetadata}\NormalTok{(}\DataTypeTok{baseUrl =}\NormalTok{ baseUrl, }
                                     \DataTypeTok{category =} \StringTok{'estimation'}\NormalTok{) }\OperatorTok{%>%}\StringTok{ }
\StringTok{  }\KeywordTok{arrange}\NormalTok{(.data}\OperatorTok{$}\NormalTok{id) }\OperatorTok{%>%}\StringTok{ }
\StringTok{  }\KeywordTok{rename_all}\NormalTok{(}\DataTypeTok{.funs =}\NormalTok{ SqlRender}\OperatorTok{::}\NormalTok{camelCaseToTitleCase) }\OperatorTok{%>%}\StringTok{ }
\StringTok{  }\KeywordTok{tail}\NormalTok{()}
\CommentTok{#> # A tibble: 6 x 7}
\CommentTok{#>   `Created Date`      `Modified Date`     `Has Write Acces~    Id Name                      Description                              Type      }
\CommentTok{#>   <dttm>              <dttm>              <lgl>             <int> <chr>                     <chr>                                    <chr>     }
\CommentTok{#> 1 2020-06-15 20:03:00 2020-06-17 02:20:00 FALSE               148 [Epi 786] comparative ef~ [EPI 786] Population-level effect estim~ Comparati~}
\CommentTok{#> 2 2020-06-19 15:27:00 2020-06-19 15:28:00 FALSE               149 COPY OF: [Epi 786] compa~ [EPI 786] Population-level effect estim~ Comparati~}
\CommentTok{#> 3 2020-08-03 18:42:00 2020-08-06 20:26:00 FALSE               153 [ONC] Blinatumomab compa~ <NA>                                     Comparati~}
\CommentTok{#> 4 2020-09-08 18:12:00 2020-09-09 16:47:00 FALSE               154 [EHDEN Academy PLE Tutor~ <NA>                                     Comparati~}
\CommentTok{#> 5 2020-09-28 17:20:00 2020-10-15 11:27:00 FALSE               155 [EPI 763] Tremfya Safety~ <NA>                                     Comparati~}
\CommentTok{#> 6 2020-10-20 18:57:00 2020-10-22 17:01:00 FALSE               156 SF Methods - Certolizuma~ <NA>                                     Comparati~}
\end{Highlighting}
\end{Shaded}

\begin{Shaded}
\begin{Highlighting}[]

\NormalTok{ROhdsiWebApi}\OperatorTok{::}\KeywordTok{getEstimationDefinitionsMetaData}\NormalTok{(}\DataTypeTok{baseUrl =}\NormalTok{ baseUrl) }\OperatorTok{%>%}\StringTok{ }
\StringTok{  }\KeywordTok{arrange}\NormalTok{(.data}\OperatorTok{$}\NormalTok{id) }\OperatorTok{%>%}\StringTok{ }
\StringTok{  }\KeywordTok{rename_all}\NormalTok{(}\DataTypeTok{.funs =}\NormalTok{ SqlRender}\OperatorTok{::}\NormalTok{camelCaseToTitleCase) }\OperatorTok{%>%}\StringTok{ }
\StringTok{  }\KeywordTok{tail}\NormalTok{()}
\CommentTok{#> # A tibble: 6 x 7}
\CommentTok{#>   `Created Date`      `Modified Date`     `Has Write Acces~    Id Name                      Description                              Type      }
\CommentTok{#>   <dttm>              <dttm>              <lgl>             <int> <chr>                     <chr>                                    <chr>     }
\CommentTok{#> 1 2020-06-15 20:03:00 2020-06-17 02:20:00 FALSE               148 [Epi 786] comparative ef~ [EPI 786] Population-level effect estim~ Comparati~}
\CommentTok{#> 2 2020-06-19 15:27:00 2020-06-19 15:28:00 FALSE               149 COPY OF: [Epi 786] compa~ [EPI 786] Population-level effect estim~ Comparati~}
\CommentTok{#> 3 2020-08-03 18:42:00 2020-08-06 20:26:00 FALSE               153 [ONC] Blinatumomab compa~ <NA>                                     Comparati~}
\CommentTok{#> 4 2020-09-08 18:12:00 2020-09-09 16:47:00 FALSE               154 [EHDEN Academy PLE Tutor~ <NA>                                     Comparati~}
\CommentTok{#> 5 2020-09-28 17:20:00 2020-10-15 11:27:00 FALSE               155 [EPI 763] Tremfya Safety~ <NA>                                     Comparati~}
\CommentTok{#> 6 2020-10-20 18:57:00 2020-10-22 17:01:00 FALSE               156 SF Methods - Certolizuma~ <NA>                                     Comparati~}
\end{Highlighting}
\end{Shaded}

This is a generic framework that applies to most WebApi categories, and
supports different types of CRUD functionalities like
deleteConceptSetDefinition() vs deleteDefinition(category =
`conceptSet').

\hypertarget{concept-set}{%
\section{Concept Set}\label{concept-set}}

Please review
\href{https://ohdsi.github.io/TheBookOfOhdsi/Cohorts.html\#conceptSets}{`Concept
sets - The Book of OHDSI'}

We commonly post concept set expression into WebApi/Atlas, or try get an
expression from Atlas/WebApi based on a conceptSetDefinitionId.

Example: lets say we have concept set expression as follows, that is
being used for a Rheumatoid Arthritis study.

\begin{Shaded}
\begin{Highlighting}[]

\NormalTok{jsonExpression <-}\StringTok{ '\{}
\StringTok{  "items": [}
\StringTok{    \{}
\StringTok{      "concept": \{}
\StringTok{        "CONCEPT_ID": 81097,}
\StringTok{        "CONCEPT_NAME": "Feltys syndrome",}
\StringTok{        "STANDARD_CONCEPT": "S",}
\StringTok{        "STANDARD_CONCEPT_CAPTION": "Standard",}
\StringTok{        "INVALID_REASON": "V",}
\StringTok{        "INVALID_REASON_CAPTION": "Valid",}
\StringTok{        "CONCEPT_CODE": "57160007",}
\StringTok{        "DOMAIN_ID": "Condition",}
\StringTok{        "VOCABULARY_ID": "SNOMED",}
\StringTok{        "CONCEPT_CLASS_ID": "Clinical Finding"}
\StringTok{      \},}
\StringTok{      "isExcluded": true,}
\StringTok{      "includeDescendants": false,}
\StringTok{      "includeMapped": false}
\StringTok{    \},}
\StringTok{    \{}
\StringTok{      "concept": \{}
\StringTok{        "CONCEPT_ID": 80809,}
\StringTok{        "CONCEPT_NAME": "Rheumatoid arthritis",}
\StringTok{        "STANDARD_CONCEPT": "S",}
\StringTok{        "STANDARD_CONCEPT_CAPTION": "Standard",}
\StringTok{        "INVALID_REASON": "V",}
\StringTok{        "INVALID_REASON_CAPTION": "Valid",}
\StringTok{        "CONCEPT_CODE": "69896004",}
\StringTok{        "DOMAIN_ID": "Condition",}
\StringTok{        "VOCABULARY_ID": "SNOMED",}
\StringTok{        "CONCEPT_CLASS_ID": "Clinical Finding"}
\StringTok{      \},}
\StringTok{      "isExcluded": false,}
\StringTok{      "includeDescendants": true,}
\StringTok{      "includeMapped": false}
\StringTok{    \},}
\StringTok{    \{}
\StringTok{      "concept": \{}
\StringTok{        "CONCEPT_ID": 4035611,}
\StringTok{        "CONCEPT_NAME": "Seropositive rheumatoid arthritis",}
\StringTok{        "STANDARD_CONCEPT": "S",}
\StringTok{        "STANDARD_CONCEPT_CAPTION": "Standard",}
\StringTok{        "INVALID_REASON": "V",}
\StringTok{        "INVALID_REASON_CAPTION": "Valid",}
\StringTok{        "CONCEPT_CODE": "239791005",}
\StringTok{        "DOMAIN_ID": "Condition",}
\StringTok{        "VOCABULARY_ID": "SNOMED",}
\StringTok{        "CONCEPT_CLASS_ID": "Clinical Finding"}
\StringTok{      \},}
\StringTok{      "isExcluded": false,}
\StringTok{      "includeDescendants": true,}
\StringTok{      "includeMapped": false}
\StringTok{    \}}
\StringTok{  ]}
\StringTok{\}'}
\end{Highlighting}
\end{Shaded}

Lets call this concept set expression - `{[}ROhdsiWebApi Vignette{]}
Rheumatoid Arthritis concept set'.

We will need to check if there is a concept set by this name.

\begin{Shaded}
\begin{Highlighting}[]
\CommentTok{# check if there is a concept set by this name, if yes, delete it}
\NormalTok{exists <-}\StringTok{ }\NormalTok{ROhdsiWebApi}\OperatorTok{::}\KeywordTok{existsConceptSetName}\NormalTok{(}\DataTypeTok{conceptSetName =}\NormalTok{ conceptSetName, }\DataTypeTok{baseUrl =}\NormalTok{ baseUrl)}
\NormalTok{exists}
\CommentTok{#> # A tibble: 1 x 5}
\CommentTok{#>   hasWriteAccess    id name                                                     createdDate         modifiedDate       }
\CommentTok{#>   <lgl>          <int> <chr>                                                    <dttm>              <dttm>             }
\CommentTok{#> 1 FALSE          11928 [ROhdsiWebApi Vignette] Rheumatoid Arthritis concept set 2020-10-06 11:14:00 2020-10-06 11:14:00}
\end{Highlighting}
\end{Shaded}

If there is a concept set with this name, then we could either choose
another name - or delete the old concept set. For this vignette we have
chosen to delete any matching concept set as follows:

\begin{Shaded}
\begin{Highlighting}[]
\ControlFlowTok{if}\NormalTok{ (}\OperatorTok{!}\KeywordTok{isFALSE}\NormalTok{(exists)) \{}
\NormalTok{  ROhdsiWebApi}\OperatorTok{::}\KeywordTok{deleteConceptSetDefinition}\NormalTok{(}\DataTypeTok{conceptSetId =}\NormalTok{ exists}\OperatorTok{$}\NormalTok{id, }\DataTypeTok{baseUrl =}\NormalTok{ baseUrl)}
\NormalTok{\}}
\CommentTok{#> Successfully deleted conceptSet definition id 11928. Request status code: Success: (204) No Content}
\CommentTok{#> NULL}
\end{Highlighting}
\end{Shaded}

Now we need to ensure the JSON expression above is converted to R-data
object. Note: By design, ROhdsiWebAPi does not accept JSON. It needs to
be converted to R (list) expression

We can now post this R-object into WebApi as follows:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{returnFromPostRequest <-}\StringTok{ }\NormalTok{ROhdsiWebApi}\OperatorTok{::}\KeywordTok{postConceptSetDefinition}\NormalTok{(}\DataTypeTok{baseUrl =}\NormalTok{ baseUrl, }
                                                                \DataTypeTok{conceptSetDefinition =}\NormalTok{ rExpression, }
                                                                \DataTypeTok{name =}\NormalTok{ conceptSetName)}
\CommentTok{#> Post ConceptSet definition was successful}
\end{Highlighting}
\end{Shaded}

If successful, we will get a return object as follows into R.

\begin{verbatim}
#> # A tibble: 1 x 5
#>   createdDate         modifiedDate        hasWriteAccess    id name
#>   <dttm>              <dttm>              <lgl>          <int> <chr>
#> 1 2020-11-25 16:26:00 2020-11-25 16:26:00 FALSE          12294 [ROhdsiWebApi Vignette] Rheumatoid Arthritis concept set
\end{verbatim}

The id of the newly posted concept-set definition is 12294. We can now
use this concept-set for many concept set queries eg.,

\hypertarget{if-want-to-print-ready-expression-of-the-concept-set-definition}{%
\subsection{if want to print ready expression of the concept set
definition}\label{if-want-to-print-ready-expression-of-the-concept-set-definition}}

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{conceptSetDefinition =}\StringTok{ }\KeywordTok{getConceptSetDefinition}\NormalTok{(}\DataTypeTok{conceptSetId =}\NormalTok{ returnFromPostRequest}\OperatorTok{$}\NormalTok{id, }
                                               \DataTypeTok{baseUrl =}\NormalTok{ baseUrl)}
\NormalTok{conceptTbl <-}
\StringTok{  }\KeywordTok{convertConceptSetDefinitionToTable}\NormalTok{(conceptSetDefinition)}
\KeywordTok{names}\NormalTok{(conceptTbl) <-}
\StringTok{  }\NormalTok{SqlRender}\OperatorTok{::}\KeywordTok{camelCaseToTitleCase}\NormalTok{(}\KeywordTok{names}\NormalTok{(conceptTbl))}
\NormalTok{conceptTbl}
\CommentTok{#> # A tibble: 3 x 13}
\CommentTok{#>   `Is Excluded` `Include Descen~ `Include Mapped` `Concept Id` `Concept Name` `Standard Conce~ `Standard Conce~ `Invalid Reason`}
\CommentTok{#>   <lgl>         <lgl>            <lgl>                   <int> <chr>          <chr>            <chr>            <chr>           }
\CommentTok{#> 1 TRUE          FALSE            FALSE                   81097 Felty's syndr~ S                Standard         V               }
\CommentTok{#> 2 FALSE         TRUE             FALSE                   80809 Rheumatoid ar~ S                Standard         V               }
\CommentTok{#> 3 FALSE         TRUE             FALSE                 4035611 Seropositive ~ S                Standard         V               }
\CommentTok{#> # ... with 5 more variables: `Invalid Reason Caption` <chr>, `Concept Code` <chr>, `Domain Id` <chr>, `Vocabulary Id` <chr>, `Concept Class}
\CommentTok{#> #   Id` <chr>}
\end{Highlighting}
\end{Shaded}

\texttt{createConceptSetWorkbook} maybe used to create an Excel workbook
of the concept set.

If we want a list of all conceptId's (including descendants) from the
concept set expression

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{resolvedConcepts =}\StringTok{ }\KeywordTok{resolveConceptSet}\NormalTok{(}\DataTypeTok{conceptSetDefinition =}\NormalTok{ conceptSetDefinition, }\DataTypeTok{baseUrl =}\NormalTok{ baseUrl)}
\KeywordTok{print}\NormalTok{(}\StringTok{"Note: Showing only the first 10 concept id's"}\NormalTok{)}
\CommentTok{#> [1] "Note: Showing only the first 10 concept id's"}
\NormalTok{resolvedConcepts[}\DecValTok{1}\OperatorTok{:}\DecValTok{10}\NormalTok{]}
\CommentTok{#>  [1]   80809 4035427 4035611 4103516 4114439 4114440 4114441 4114442 4114444 4115050}
\end{Highlighting}
\end{Shaded}

The concept set expression json expression can be recaptured from WebApi
as follows

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{json <-}
\StringTok{  }\KeywordTok{getConceptSetDefinition}\NormalTok{(}\DataTypeTok{baseUrl =}\NormalTok{ baseUrl, }
                          \DataTypeTok{conceptSetId =}\NormalTok{ returnFromPostRequest}\OperatorTok{$}\NormalTok{id}
\NormalTok{                          )}\OperatorTok{$}\NormalTok{expression }\OperatorTok{%>%}\StringTok{ }
\StringTok{  }\NormalTok{RJSONIO}\OperatorTok{::}\KeywordTok{toJSON}\NormalTok{(}\DataTypeTok{pretty =} \OtherTok{TRUE}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

Similar framework maybe used with other WebApi categories such as
Cohorts/Characterization/Incidence Rate.

\hypertarget{applications-of-rohdsiwebapi}{%
\section{Applications of
ROhdsiWebAPi}\label{applications-of-rohdsiwebapi}}

A valuable feature of ROhdsiWebApi is that it is able to get full result
set into R, as a data frame object. Results of Cohort, Characterization,
Incidence Rate, Pathway maybe obtained. This data frame may then be
converted to publication ready material by using packages like
\href{https://davidgohel.github.io/officer/}{Officer},
\href{https://davidgohel.github.io/flextable/}{flextable}. The functions
in ROhdsiWebAPi maybe used to create dynamic R-shiny apps that allow
user to interact with WebApi and select cohort definitions, concept sets
for review or modifications. ROhdsiWebApi may be used to build `mini
versions' of Atlas that is project specific - by directly interacting
with WebApi using R.

\hypertarget{cohortscharacterizationincidence-rate}{%
\subsection{Cohorts/Characterization/Incidence
rate}\label{cohortscharacterizationincidence-rate}}

Please review
\href{https://ohdsi.github.io/TheBookOfOhdsi/Cohorts.html\#what-is-a-cohort}{`What
is a cohort - The Book of OHDSI'}.

\begin{quote}
We define a cohort as a set of persons who satisfy one or more inclusion
criteria for a duration of time. The term cohort is often interchanged
with the term phenotype. Cohorts are used throughout OHDSI analytical
tools and network studies as the primary building blocks for executing a
research question.
\end{quote}

\begin{quote}
A cohort is defined as the set of persons satisfying one or more
inclusion criteria for a duration of time. One person may qualify for
one cohort multiple times during non-overlapping time intervals. Cohorts
are constructed in ATLAS by specifying cohort entry criteria and cohort
exit criteria. Cohort entry criteria involve selecting one or more
initial events, which determine the start date for cohort entry, and
optionally specifying additional inclusion criteria which filter to the
qualifying events. Cohort exit criteria are applied to each cohort entry
record to determine the end date when the person's episode no longer
qualifies for the cohort.
\end{quote}

Cohorts/Characterization/Incidence Rate are WebApi categories, where
WebApi manages the execution of generations.

Example: We may want to know if a certain cohort specification has been
generated by checking cohort generation status
\texttt{getCohortGenerationInformation(baseUrl\ =\ baseUrl,\ cohortId=\ 4234)}.
If a cohort is not previously generated, it may be generated using
\texttt{invokeCohortSetGeneration(baseUrl\ =\ baseUrl,\ cohortId\ =\ 4234,\ sourceKey\ =\ \textquotesingle{}HCUP\textquotesingle{})}
. If it is already generated, we can extract its output of cohort
generation using \texttt{getCohortResults(baseUrl,\ cohortId\ =\ 4234)}.

\hypertarget{characterization}{%
\subsection{Characterization}\label{characterization}}

Please review
\href{https://ohdsi.github.io/TheBookOfOhdsi/Characterization.html}{`Characterization
- The Book of OHDSI'}.

\hypertarget{population-level-effect-estimation}{%
\subsection{Population Level Effect
Estimation}\label{population-level-effect-estimation}}

Please review
\href{https://ohdsi.github.io/TheBookOfOhdsi/PopulationLevelEstimation.html}{`Population
Level Effect Estimation - The Book of OHDSI'}.

\hypertarget{patient-level-prediction}{%
\section{Patient Level Prediction}\label{patient-level-prediction}}

Please review
\href{https://ohdsi.github.io/TheBookOfOhdsi/PatientLevelPrediction.html}{`Patient
Level Prediction - The Book of OHDSI'}.

\end{document}

max_stars_repo_name: azimov/ROhdsiWebApi

max_issues_repo_name: azimov/ROhdsiWebApi

max_forks_repo_name: azimov/ROhdsiWebApi


text:
\subsubsection{Interfaced}
\label{Interfaced}
The \textbf{Interfaced} PostProcessor allows the user
to create its own Post-Processor. While the External PostProcessor (see
Section~\ref{External} allows the user to create case-dependent
Post-Processors, with this new class the user can create new general
purpose Post-Processors.
%

\ppType{Interfaced}{Interfaced}

\begin{itemize}
  \item \xmlNode{method}, \xmlDesc{comma separated string, required field},
  lists the method names of a method that will be computed (each
  returning a post-processing value). All available methods need to be included
  in the ``/raven/framework/PostProcessorFunctions/'' folder
\end{itemize}

\textbf{Example:}
\begin{lstlisting}[style=XML,morekeywords={subType,debug,name,class,type}]
<Simulation>
  ...
  <Models>
    ...
    <PostProcessor name="example" subType='InterfacedPostProcessor'verbosity='debug'>
       <method>testInterfacedPP</method>
       <!--Here, the xml nodes required by the chosen method have to be
       included.
        -->
    </PostProcessor>
    ...
  </Models>
  ...
</Simulation>
\end{lstlisting}

All the \textbf{Interfaced} PostProcessors need to be contained in the
``/raven/framework/PostProcessorFunctions/'' folder. In fact, once the
\textbf{Interfaced} post-processor is defined in the RAVEN input file, RAVEN
search that the method of the post-processor is located in such folder.

The class specified in the \textbf{Interfaced} post-processor has to inherit the
PostProcessorInterfaceBase class and the user must specify this set of
methods:
\begin{itemize}
  \item initialize: in this method, the internal parameters of the
  post-processor are initialized. Mandatory variables that needs to be
  specified are the following:
\begin{itemize}
  \item self.inputFormat: type of dataObject expected in input
  \item self.outputFormat: type of dataObject generated in output
\end{itemize}
  \item readMoreXML: this method is in charge of reading the PostProcessor xml
  node, parse it and fill the PostProcessor internal variables.
  \item run: this method performs the desired computation of the dataObject.
\end{itemize}

\begin{lstlisting}[language=python]
from PostProcessorInterfaceBaseClass import PostProcessorInterfaceBase
class testInterfacedPP(PostProcessorInterfaceBase):
  def initialize(self)
  def readMoreXML(self,xmlNode)
  def run(self,inputDic)
\end{lstlisting}

\paragraph{Data Format}
The user is not allowed to modify directly the DataObjects, however the
content of the DataObjects is available in the form of a python dictionary.
Both the dictionary give in input and the one generated in the output of the
PostProcessor are structured as follows:

\begin{lstlisting}[language=python]
inputDict = {'data':{}, 'metadata':{}}
\end{lstlisting}

where:

\begin{lstlisting}[language=python]
inputDict['data'] = {'input':{}, 'output':{}}
\end{lstlisting}

In the input dictionary, each input variable is listed as a dictionary that
contains a numpy array with its own values as shown below for a simplified
example

\begin{lstlisting}[language=python]
inputDict['data']['input'] = {'inputVar1': array([ 1.,2.,3.]),
                              'inputVar2': array([4.,5.,6.])}
\end{lstlisting}

Similarly, if the dataObject is a PointSet then the output dictionary is
structured as follows:

\begin{lstlisting}[language=python]
inputDict['data']['output'] = {'outputVar1': array([ .1,.2,.3]),
                               'outputVar2':array([.4,.5,.6])}
\end{lstlisting}

However, if the dataObject is a HistorySet then the output dictionary is
structured as follows:

\begin{lstlisting}[language=python]
inputDict['data']['output'] = {'hist1': {}, 'hist2':{}}
\end{lstlisting}

where

\begin{lstlisting}[language=python]
inputDict['output']['data'][hist1] = {'time': array([ .1,.2,.3]),
                              'outputVar1':array([ .4,.5,.6])}
inputDict['output']['data'][hist2] = {'time': array([ .1,.2,.3]),
                              'outputVar1':array([ .14,.15,.16])}
\end{lstlisting}

ext: tex

max_stars_repo_name: FlanFlanagan/raven

max_issues_repo_name: FlanFlanagan/raven

max_forks_repo_name: FlanFlanagan/raven


text:
\documentclass[10pt]{article}

\usepackage{latexsym}
\usepackage{amssymb}
\usepackage{code}
\usepackage{graphicx}

\usepackage{listings}

\usepackage[english]{babel}
%\usepackage{blindtext}
%\usepackage{fontspec}
%\setmainfont{Arial}

%\renewcommand{\familydefault}{\sfdefault}
%\usepackage{blindtext}
\pagenumbering{arabic}  % Arabic page numbers GM July 2000

\usepackage{authoraftertitle}
\usepackage{fancyhdr}
% Clear the header and footer
\fancyhead{}
\fancyfoot{}
%\lhead{\includegraphics[height=0.7cm]{../logo/nyp-logo-int.png} }
\rhead{\scriptsize \MyTitle}
%\lfoot{\scriptsize Specialist Diploma in Business \& Big Data Analytics\\ Copyright
%  \copyright\ Nanyang Polytechnic. All Rights Reserved.}
\rfoot{\thepage}
\pagestyle{fancy}

\renewcommand{\topfraction}{0.95}
\renewcommand{\textfraction}{0.02}
\renewcommand{\floatpagefraction}{0.95}

\bibliographystyle{plain}

%%\textwidth      150mm \textheight     210mm \oddsidemargin  -2mm \evensidemargin -2mm

\renewcommand{\baselinestretch}{0.987}

\setlength{\parskip}{0.0in}

\newcommand{\Nturns}{\, \vdash_{\mbox{\scriptsize lnf}} \,}

\newcommand{\tr}[1]{}
%%\newcommand{\tr}[1]{#1}
%%\newcommand{\nottr}[1]{#1}
\newcommand{\nottr}[1]{}

%\newcommand{\implies}{\supset}
\newcommand{\clabel}[1]{\mbox{(#1)}}
\newcommand{\rat}[1]{\rightarrowtail_{#1}}
\newcommand{\arr}{\rightarrow}
\newcommand{\arrow}{\rightarrow}
\newcommand{\Arr}{\Rightarrow}
\newcommand{\atsign}{@}
\newcommand{\simparrow}[0]{\Longleftrightarrow}
\newcommand{\proparrow}[0]{\Longrightarrow}
\newcommand{\comment}[1]{}
\newcommand{\ignore}[1]{}
%\newcommand{\kl}[1]{{\bf KL:#1}} %%%{\marginpar{\sc kl}{\bf #1}}
\newcommand{\kl}[1]{}
\newcommand{\ms}[1]{{\bf MS:#1}}       %%{\marginpar{\sc ms}{\bf #1}}
%%\newcommand{\jw}[1]{\marginpar{\sc jw}{\bf #1}}
%% \newcommand{\answer}[1]{#1}

\newcommand{\pjs}[1]{}
%%\newcommand{\ms}[1]{}
\newcommand{\jw}[1]{}
%%\newcommand{\kl}[1]{}

\newcommand{\pow}{\^{}}
\newcommand{\venv}{\Delta}
\newcommand{\mleq}{\mbox{\tt leq}}
\newcommand{\mmleq}{\leq}
\newcommand{\mas}{\mbox{\tt as}}
\newcommand{\mfix}{\mu}

\newcommand{\mysection}[1]{\vspace*{-2mm}\section{#1}\vspace*{-1mm}}
\newcommand{\mysubsection}[1]{\vspace*{-1mm}\subsection{#1}\vspace*{-1mm}}

\newcounter{cnt}
\newtheorem{ex}{Example}
%\newenvironment{example}{
%        \begin{ex}\rm}%
%    {\hfill$\Box$\end{ex}}

\newenvironment{nexample}{
        \begin{ex}\rm}%
        {\end{ex}}

%%\newtheorem{rem}{Remark}
%%\renewenvironment{remark}{
%%        \begin{rem}\rm}%
%%    {\hfill$\Box$\end{rem}}

%%\newenvironment{myexample}{
%%        \begin{ex}\rm}%
%%  {\hfill $\Diamond$\end{ex}}
%\newenvironment{example}{
%        \begin{example}\rm}%
%    {\end{example}}

\newenvironment{ttline}{\begin{trivlist}\item \tt}{\end{trivlist}}
\newenvironment{ttprog}{\begin{trivlist}\small\item \tt
        \begin{tabbing}}{\end{tabbing}\end{trivlist}}

\newcommand{\figcode}[1]
        {\begin{figure*}[t]#1
        \end{figure*}}

\title{%ITD353 Massively Parallel Computing for Big Data \\
  Introduction to Scala }
%\author{Kenny Zhuo Ming Lu\\
%  \multicolumn{1}{p{.7\textwidth}}{\centering
%  \emph{School of Information Technology \\ Nanyang Polytechnic \\
%    180 Ang Mo Kio Avenue 8, Singapore 569830}}
%}

\begin{document}
\maketitle \makeatactive
\thispagestyle{fancy}

\lstset{language=Python}

%\begin{abstract}
%\end{abstract}

\section{Learning Outcomes} \label{sec:aims}
\begin{itemize}
 \item Start Scala REPL in Scala application development
 \item Execute and observe Scala programs Scala application
   development
 \item Comprehend all the Scala languages features and the program semantics when reviewing Scala source codes
 \item Develop data transformation scripts using Scala
\end{itemize}

\section{Scala Features}
\begin{enumerate}
\item Scala is an \underline{o~~~~~~~~~~~~~~} oriented and \underline{f~~~~~~~~~~~~~~} language.
\item Scala is a \underline{~~~~~~~~~~~~~~} typed language.
\end{enumerate}

\section{First Scala Program - Hello World}
\begin{enumerate}
\item Check out the source codes.
  \begin{enumerate}
  \item go to Github and download the {\tt scala.zip} from
\end{enumerate}
\begin{code}
      $ cd learning-scala/codes
\end{code}
\item Examine the script {\tt Script.scala}  in {\tt helloworld}.
\item Execute the script with the following
\begin{code}
$ scala Script.scala
\end{code}
\item Examine the code {\tt Main.scala} in {\tt helloworld}.
\item Compile the code
\begin{code}
$ scalac Main.scala
\end{code}
\item Execute the compiled code
\begin{code}
$ scala Main
\end{code}
\end{enumerate}

\section{Scala REPL}
\begin{enumerate}
\item Start a terminal in Linux or command line in Windows, type
 \begin{code}
  $ scala
  \end{code}
 Note that the \$ sign is the command prompt, you should not include it as
 part of the command.
\item Exit Scala REPL by typing
 \begin{code}
 scala> :quit
 \end{code}
 Note that the {\tt scala>} sign is the Scala REPL prompt, you should not include it as
 part of the command.
\end{enumerate}
%$

\section{Variables, Values and Assignment Statement}
In a Scala REPL
\begin{enumerate}
\item Declare a variable with name ``first\_name'' and assign a string
  value as ``robin''.
\item Declare a value with name ``last\_name'' and assign a string
  value as ``Williams''.
\item Update the variable ``first\_name'' to a new string value
  ``Robin''
\item If you were to update the value ``last\_name'' to a new string
  ``Hood'', what will happen?
\end{enumerate}
\comment{
\subsection{Sample Answer}
\begin{code}
scala> var first_name = "robin"
first_name: String = robin

scala> val last_name = "Williams"
last_name: String = Williams

scala> first_name = "Robin"
first_name: String = Robin

scala> last_name = "Hood"
<console>:12: error: reassignment to val
       last_name = "Hood"
                 ^
\end{code}
}
\section{Print Statement}
In a Scala REPL
\begin{enumerate}
\item Print the variable ``first\_name'' and value ``last\_name'' individually
\item Use template, print the following
\begin{code}
Robin William (1951 - 2014)
\end{code}
You need to make use of the variable ``first\_name'' and value ``last\_name'',
and put 1951 and 2014 into the two additional variables. For instance,
assuming you have defined ``first\_name'' and ``last\_name''.
\begin{code}
val bYear = 1951
val dYear = 2014
println(s"$first_name $last_name ($bYear - $dYear)")
\end{code}

\end{enumerate}
\comment{
\subsection{Sample Answer}
\begin{code}
scala>  val bYear = 1951
bYear: Int = 1951

scala>  val dYear = 2014
dYear: Int = 2014

scala> println(s"$first_name $last_name ($bYear - $dYear)")
Robin Williams (1951 - 2014)
\end{code}
}

\section{If-else}

\begin{enumerate}
\item Type the following code snippet in the Scala REPL and observe the output.
\begin{code}
val i = 1
if (i / 2 >= 0.5) {
  println(s" ${i} / 2 is greater than or equal to  0.5") }
else {
   println(s"${i} / 2 is less than 0.5")
}
\end{code}
\end{enumerate}

\section{List and List operation}

\begin{enumerate}
\item Declare a list of integer {\tt l1} with integers 1, 2, 3 and 4.
\item Declare a second list {\tt l2} whose elements are the odd values of {\tt
    l1} incremented by 1.
\item Find out the head and the tail of {\tt l2}.
\item Reverse {\tt l2}.
\item Concatenate {\tt l1} and {\tt l2}
\item Compute the sum of {\tt l1}
\end{enumerate}

\comment{
\subsection{Sample Answer}
\begin{code}
scala> val l1 = List(1,2,3,4)
l1: List[Int] = List(1, 2, 3, 4)

scala> val l2 = for { x <- l1; if x \% 2 == 1} yield (x + 1)// or
l2: List[Int] = List(2, 4)

scala> val l2a = l1.filter ( x => x \% 2 == 1 ).map( x => x + 1)
l2a: List[Int] = List(2, 4)

scala> l2.head
res0: Int = 2

scala> l2.tail
res1: List[Int] = List(4)

scala> l2.reverse
res2: List[Int] = List(4, 2)

scala> l1 ++ l2
res3: List[Int] = List(1, 2, 3, 4, 2, 4)

scala> l1.foldLeft(0)( (s,x) => s + x )
res4: Int = 10
\end{code}
}

\section{Object Oriented Programming}

\begin{enumerate}
\item In the terminal, change the working directory to {\tt
    {}~/git/learning-scala/codes/oop}.
\item Examine the code {\tt OOP.scala}, are you able to identify the
  class constructors, member fields, member methods? Are you able to
  identify the class inheritence?
\begin{code}
class Person(n:String,i:String) {
	private val name:String = n
	private val id:String   = i
	def getName():String = name
	def getId():String = id
}

trait NightOwl {
	def stayUpLate():Unit
}

class Student(n:String, i:String, g:Double) extends Person(n,i) with NightOwl {
	private var gpa = g
	def getGPA() = gpa
	def setGPA(g:Double) =
	{
		gpa = g
	}
	override def stayUpLate():Unit =
	{
		println("woohoo")
	}
}

class Staff(n:String, i:String, sal:Double) extends Person(n,i) {
	private var salary = sal
	def getSalary() = salary
	def setSalary(sal:Double) =
	{
		salary = sal
	}
}
\end{code}
\item Load the class in the Scala REPL and test it out
\begin{code}
scala> :load OOP.scala
Loading OOP.scala...
defined class Person
defined trait NightOwl
defined class Student
defined class Staff

scala> val tom = new Student("Tom", "X1235", 4.0)
tom: Student = Student@601c1dfc

scala> val jerry = new Staff("Jerry", "T0001", 500000.0)
jerry: Staff = Staff@650fbe32

scala> tom.stayUpLate
woohoo
\end{code}
\end{enumerate}

\section{Functional Programming in Scala}
\begin{enumerate}
\item In the terminal, change the working directory to {\tt
    {}~/git/learning-scala/codes/fp}.
\item Examine the code {\tt Exp.scala}, are you able to identify the
  sealed trait, the case class, and the pattern matching?
\begin{code}
sealed trait Exp
case class Val(v:Int) extends Exp
case class Plus(e1:Exp, e2:Exp) extends Exp

def simp(e:Exp):Exp = e match
{
	case Val(v) => e
	case Plus(Val(0), e2) => e2
	case Plus(e1,e2) => Plus(simp(e1), simp(e2))
}
\end{code}

\item Run it with Scala REPL
\begin{code}
$ scala
scala> :load Exp.scala
scala> val e = Plus(Val(0), Plus(Val(1), Val(2)))
e: Plus = Plus(Val(0),Plus(Val(1),Val(2)))

scala> simp(e)
res0: Exp = Plus(Val(1),Val(2))
\end{code}
%$
\item Note that $x - 0 = x$, $x * 1 = x$, $x / 1 = x$ for all $x$, can
  we extend our {\tt Exp} data type and the simplification {\tt simp} to handle minus, multiplication, and division?
\item Note that the simplification is not througout, e.g.
\begin{code}
scala> val e2 = Plus(Val(0), Plus(Val(0),Val(2)))
e2: Plus = Plus(Val(0),Plus(Val(0),Val(2)))

scala> simp(e2)
res1: Exp = Plus(Val(0),Val(2))
\end{code}
How can we fix it?
\end{enumerate}

\comment{
\subsection{Sample Answer}

\begin{code}
sealed trait Exp
case class Val(v:Int) extends Exp
case class Plus(e1:Exp, e2:Exp) extends Exp
case class Minus(e1:Exp, e2:Exp) extends Exp
case class Mult(e1:Exp, e2:Exp) extends Exp
case class Div(e1:Exp, e2:Exp) extends Exp

def simp(e:Exp):Exp = e match
{
        case Val(v)            => e
        case Plus(Val(0), e2)  => e2
        case Plus(e1, Val(0))  => e1
        case Plus(e1,e2)       => Plus(simp(e1), simp(e2))
        case Minus(e1, Val(0)) => e1
        case Minus(e1, e2)     => Minus(simp(e1), simp(e2))
        case Mult(Val(1), e2)  => e2
        case Mult(e1, Val(1))  => e1
        case Mult(e1, e2)      => Mult(simp(e1), simp(e2))
        case Div(e1, Val(1))   => e1
        case Div(e1, e2)       => Div(simp(e1), simp(e2))
}

def simpFix(e:Exp):Exp =
{
        val e2 = simp(e)
        if (e == e2)
        {
                e
        }
        else
        {
                simpFix(e2)
        }
}
\end{code}
}

%\item Implement An Interpreter of Lambda Calculus

\section{Data Processing}

\subsection{CSV}

Given a CSV file as follows,
\begin{code}
user1,account1,230
user2,account2,130
user3,account3,40
user4,account4,34
user5,account5,23
user6,account6,20
user7,account7,199
user8,account8,324
user9,account9,159
user10,account10,123
\end{code}

The following Scala code loads the CSV file from the file system and
replaces the commas by ``$|$'' , prints out the lines on the standard
output and writes the lines into an output file

\begin{code}
object DataProc extends App {
  import java.io._
  val inFile = io.Source.fromFile("./input.csv")
  val outFile = new PrintWriter(new File("./output.txt" ))
  for (line <- inFile.getLines) {
    val cols = line.split(",").map(_.trim)
    println(s"${cols(0)}|${cols(1)}|${cols(2)}")
    outFile.write(s"${cols(0)}|${cols(1)}|${cols(2)}\n")
  }
  inFile.close
  outFile.close
}
\end{code}

The output.txt will looks like
\begin{code}
user1|account1|230
user2|account2|130
user3|account3|40
user4|account4|34
user5|account5|23
user6|account6|20
user7|account7|199
user8|account8|324
user9|account9|159
user10|account10|123
\end{code}

\begin{enumerate}
 \item  Modify the above code to find and write the min, max, mean,
   median of the account balance (the third column). The output.txt
   should look like
\begin{code}
min: 20
max: 324
mean: 128
median: 126
\end{code}
  Hint: to convert a string value into integer, you may use the {\tt
    toInt} method
  following
\begin{code}
scala> val x = "1"
x: String = 1
scala> x.toInt
res0: Int = 1
\end{code}
 \item In the last task, if your implementation of finding the min,
   max, mean and median requires the use of for-loop, can you rewrite
   it using {\tt map}, {\tt sortWith} and {\tt foldLeft}?
\end{enumerate}

\subsection{JSON}

JSON is another popular data type format. Many libraries allow us to map a relation between a
Scala data structure and a JSON object. One of them is {\tt gson}.

The following example code demonstrates how a Scala case class {\tt
  Contact} (A phone contact) be converted to and from JSON objects.

\begin{code}
import com.google.gson.Gson

object JSONExample extends App {
    case class Contact(firstName: String, lastName: String, phone: Int)
    val ironman = new Contact("Tony", "Stark", 12345678)
    val gson = new Gson
    println(gson.toJson(ironman))
    val cap = gson.fromJson("""
        {"firstName":"Steve","lastName":"Roger","phone":0}
    """, classOf[Contact]
    )
    println(gson.toJson(cap))
}
\end{code}

Using {\tt scala} directly to interpret the {\tt
  JSONExample.scala} file won't work, because gson demands the existence of the
class for conversion, (in this case, the class {\tt Contact} needs to
be compiled first before the conversion taking place. We first compile
then run
\begin{code}
  $ scalac JSONExample.scala  -cp gson-2.3.1.jar
  $ scala -cp ./gson-2.3.1.jar:. JSONExample
\end{code}
We see the following output
\begin{code}
{"firstName":"Tony","lastName":"Stark","phone":12345678}
{"firstName":"Steve","lastName":"Roger","phone":0}
\end{code}

Based on the example above, write a Scala program to turn the
  following JSON file
  \begin{code}
{
   "library": [
      {
         "id":"01",
         "language": "Java",
         "edition": "third",
         "author": "Herbert Schildt"
      },
      {
         "id":"07",
         "language": "C++",
         "edition": "second",
         "author": "E.Balagurusamy"
      }
   ]
}
  \end{code}
  Into a CSV file
  \begin{code}
    01,Java,third,Herbert Schildt
    07,C++,second,E.Balagurursamy
  \end{code}

%
Hint, you may considering using the following data type to transform
the JSON object into CSV format
\begin{code}
    case class Book(id:String, language:String, edition:String, author:String)
    case class Library(books:Array[Book])
  \end{code}

Other implementations of JSON conversion in Scala can be found in the
following
\begin{enumerate}
\item
\begin{verbatim}
https://app.assembla.com/spaces/liftweb/wiki/JSON_Support
\end{verbatim}
\item
\begin{verbatim}
https://www.playframework.com/documentation/2.6.x/ScalaJson
\end{verbatim}
\end{enumerate}

\subsection{Regex}

Regular expression is a powerful embeded language to match extract
sequence of items, for instance strings. For instance the following
regex extract all anchored link from a HTML page.
\begin{code}
]*>
\end{code}
Note that the sub pattern
\begin{code}
  href=\"([^\"]*)\"
\end{code}
performs most of the extraction. The parantheses define a capturing group which captures and stores the
link for output usage. The sub pattern
\begin{code}
 [\^"]*
\end{code}
matches string of any length as
long as it does not contain the double quote.

By embedding the above regex in our Scala code as follows, we are able
to extract all links in a HTML page.
\begin{code}
import java.io._
object LinkExtractor extends App {

    val inFile = io.Source.fromFile("./page.html")
    val outFile = new PrintWriter(new File("./output.txt" ))
    val anchor = """.*]*>.*""".r
    val links = for (line <- inFile.getLines)  {
        line match {
            case anchor(link) =>
                outFile.write(s"${link}\n")
            case _ => ()
        }
    }
    inFile.close
    outFile.close
}
\end{code}
%$

Note that there is a problem with the above implementation, it only
extracts the first match in every line if there exists a match. To
explore all sub matches in a line, we need to use the {\tt findAllIn}
method.
\begin{code}

import java.io._
object LinkExtractor2 extends App {
    val inFile = io.Source.fromFile("./page.html")
    val outFile = new PrintWriter(new File("./output.txt" ))
    val anchor = """]*>""".r
    val links = for (line <- inFile.getLines) {
        for (link <- anchor.findAllIn(line)) {
            outFile.write(s"${link}\n")
        }
    }
    inFile.close
    outFile.close
}
\end{code}
%$

For all the valid syntax of regex and testing of regex, please refer to
\begin{verbatim}
https://regex101.com/
\end{verbatim}

\subsection{Stop words removal}
Stop words such as ``the'', ``a'', ``is'' are too common to make a
different in text analytics model. It is a common practice to remove
them from the corpuses before processing. The following Scala script
does the remove by refering to a predefine set of stop words.
\begin{code}
import java.io._

object StopWords extends App {
  val inFile = io.Source.fromFile("./input.txt")
  val outFile = new PrintWriter(new File("./output.txt" ))
  // TODO: read the stop words from a file.
  val stop_words = List("and", "is", "it", "are", "in", "rt", "what", "from",
                "her", "to", "their", "you", "me", "his", "http", "that", "they", "by",
                "he", "a", "on", "for", "i", "of", "this", "she", "the", "my", "at").toSet
  for (line <- inFile.getLines) {
    val words = line.toLowerCase.split(" ").map(_.trim)
    val no_stopwords = words.filter( w => !stop_words.contains(w))
    outFile.write(no_stopwords.mkString(" ") + "\n")
  }
  inFile.close
  outFile.close
}
\end{code}
%
Given the {\tt input.txt} file contains the following
\begin{code}
The quick brown fox jumps over the lazy dog
\end{code}
the {\tt output.txt} file will be generated with the following content
\begin{code}
quick brown fox jumps over lazy dog
\end{code}

Your task is to modify the above script so that the stop words are
read from a file {\tt stopwords.txt} instead of being hard-coded in
the script.

\subsection{Word count}
To generate infographic with word cloud, we have to count the number
of occurrences of all the words appearing in the corpuses.  For
instance, the word count of the following file
\begin{code}
hey diddle diddle
the cat and the fiddle
the cow jumped over the
 moon
the little dog laughed
to see such sport
and the dish ran away
 with the spoon
\end{code}

Will be
\begin{code}
,2
moon,1
jumped,1
cow,1
ran,1
dog,1
such,1
dish,1
to,1
spoon,1
sport,1
over,1
diddle,2
little,1
with,1
cat,1
see,1
hey,1
away,1
and,2
laughed,1
the,7
fiddle,1
\end{code}

There are two possible implementation of the word count. The first
implementation uses imperative style (similar to Java)
\begin{code}
import java.io._
import scala.collection._
object WordCount extends App {
  val inFile = io.Source.fromFile("./input.txt")
  val outFile = new PrintWriter(new File("./output.txt" ))
  var dict = Map[String,Int]()
  for (line <- inFile.getLines) {
    val words = line.split(" ")
    for (w <- words) {
      dict.get(w) match {
        case None => dict = dict + ((w,1))
        case Some(counts) => dict = dict + ((w,counts + 1))
      }
    }
  }
  val wordCount = dict.toList
  for ( wc <- wordCount ) {
    outFile.write(s"${wc._1},${wc._2}\n")
  }
  inFile.close
  outFile.close
}
\end{code}
The second implementation uses functional programming (map and fold)
\begin{code}
import java.io._
import scala.collection._
object WordCountFun extends App {
  val inFile = io.Source.fromFile("./input.txt")
  val outFile = new PrintWriter(new File("./output.txt" ))
  val words = inFile.getLines.toList.flatMap( line => line.split(" ") )
  val empty_dict = Map[String,List[Int]]()
  val wordCount = words.map(w => (w,1))
          .foldLeft(empty_dict)( (dict,t) => t match {
            case (w,one) => dict.get(w) match {
              case None => dict + ((w,List(one)))
              case Some(ones) => dict + ((w,ones ++ List(one)))
            }
          })
          .toList
          .map( wl => wl match {
            case (w,l) => (w,l.sum)
          })
  for ( wc <- wordCount ) {
    outFile.write(s"${wc._1},${wc._2}\n")
  }
  inFile.close
  outFile.close
}
\end{code}

\subsection{TF-IDF}

\begin{itemize}
 \item TF-IDF stands for Term frequency - Inverse document
   frequency. It is a classic and powerful technique to search for
   interesting terms from a set of documents.
 \item TF is actually the word count. For instance, consider the
   following two values found in some input text file.
   \begin{code}
      apple smart phone
      android smart phone
   \end{code}
   The term frequency is
   \begin{code}
     apple, 1
     android, 1
     phone, 2
     smart, 2
   \end{code}
 \item IDF is is the total number of documents/records divided by the
   total number of the documents/records containing the words. We
   apply logarithmic to the quotient.
  The IDF for the above example is
 \begin{code}
    apple, log(2/1)
    android, log(2/1)
    phone, log(2/2)
    smart, log(2/2)
  \end{code}
   that is
 \begin{code}
    apple, 0.693
    android, 0.693
    phone, 0
    smart, 0
  \end{code}
  \item TF-IDF is obtained by multiplying the TF with the IDF.
 \begin{code}
    apple, 0.693
    android, 0.693
    phone, 0
    smart, 0
  \end{code}
  This shows that apple and android are the interesting terms coming
  from the records.
 \item Please use Scala to develop a script to compute the
   TF-IDF of a given file containing text.
\end{itemize}

\end{document}


text:
% spell-checker: disable %
\begin{pycode}[manager_appendix]
manager_appendix = texfigure.Manager(
    pytex, './',
    number=3,
    python_dir='python',
    fig_dir='figures',
    data_dir='data',
)
\end{pycode}
% spell-checker: enable %
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%                                   Appendix 1                                %
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Non-equilibrium Ion Populations}\label{nei}

In order to account for ionization non-equilibrium, we compute $f_{X,k}$ as a function of time $t$ by solving the time-dependent ion population equations for each ion $k$ in each element $X$,
\begin{equation}\label{eq:nei}
    \frac{\partial f_k}{\partial t} = n_e(R_{k+1}f_{k+1} + I_{k-1}f_{k-1} - I_kf_k - R_kf_k)
\end{equation}
where $n_e$ is the electron density and $R_k$ and $I_k$ are the temperature-dependent recombination and ionization rates of ion $k$, respectively. The ionization and recombination rates are computed using the data provided in CHIANTI. The ionization rates include both direct ionization and excitation autoionization and the recombination rates include both radiative and dielectronic recombination \citep[see section 6 of][]{young_chianti_2016}. Setting the left hand side of \autoref{eq:nei} to zero gives the equation of ionization equilibrium.

Note that for an element with atomic number $Z$, we must solve $Z+1$ coupled differential equations to find the non-equilibrium level populations. Following the approaches of \citet{masai_x-ray_1984,hughes_self-consistent_1985,shen_lagrangian_2015}, we can write \autoref{eq:nei} in matrix form,
\begin{equation}\label{eq:nei_mat}
    \frac{\partial}{\partial t}\mathbf{F} = \mathbf{A}\mathbf{F},
\end{equation}
where $\mathbf{F}=(f_1,f_2,\ldots,f_k,\ldots,f_{Z+1})$ and $\mathbf{A}$ is a ${Z+1\times Z+1}$ tridiagonal matrix containing the ionization and recombination rates, multiplied by the electron density,
\begin{equation}\label{eq:rate_mat}
    \mathbf{A} = n_e
        \begin{pmatrix}
            -I_1 & R_2 & 0 & \dots & 0 \\
            I_1 & -(I_2 + R_2) & R_3 & \dots & 0 \\
             & \ddots & \ddots & &  \\
            \vdots & I_{k-1} & -(I_k + R_k) & R_{k+1} & \vdots \\
             & & \ddots & \ddots & \\
            0 & \dots & 0 & I_{Z} & -R_{Z+1}
        \end{pmatrix}.
\end{equation}

Due to drastic changes in the ionization and recombination rates with temperature, the above system of equations is very ``stiff,'' making explicit schemes extremely sensitive to the choice of time step \citep{macneice_numerical_1984,bradshaw_numerical_2009}. To solve \autoref{eq:nei_mat}, we use the ``deferred correction'' method of \citet{npl_modern_1961}, as pointed out by \citet{macneice_numerical_1984},
\begin{equation*}
    \mathbf{F}_{j+1} = \mathbf{F}_j + \frac{\Delta t}{2}\left(\frac{\partial}{\partial t}\mathbf{F}_{j+1} + \frac{\partial}{\partial t}\mathbf{F}_j\right) + \mathcal{O}(\Delta t^2),
\end{equation*}
where $\Delta t$ is the time step, $j$ indexes time, and $\mathcal{O}(\Delta t^2)$ denotes terms of second order or higher in $\Delta t$. Dropping the higher-order terms and using \autoref{eq:nei_mat} yields an expression for $\mathbf{F}_{j+1}$,
\begin{equation}\label{eq:nei_iterative}
    \mathbf{F}_{j+1} \approx \left(\mathbb{I} - \frac{\Delta t}{2}\mathbf{A}_{j+1}\right)^{-1}\left(\mathbb{I} + \frac{\Delta t}{2}\mathbf{A}_{j}\right)\mathbf{F}_j,
\end{equation}
where $\mathbb{I}$ is the identity matrix. To solve \autoref{eq:nei_mat}, we need only compute $\mathbf{A}_j$ for each $T(t_j)$, set $\mathbf{F}_0$ to the equilibrium ion populations, and iteratively compute \autoref{eq:nei_iterative} for all $j$. We solve \autoref{eq:nei_mat} for all elements in \autoref{tab:elements} and for each strand in the \AR{}.

Though this method is unconditionally stable, $\Delta t$ should still be chosen carefully as $\mathbf{F}$ will relax to the equilibrium charge states for very long time steps. In general, smaller time steps should be chosen when the electron temperature varies rapidly and the electron density is high. In this paper, we exploit the adaptive time step provided by the two-fluid EBTEL code which accounts for changes in the temperature and electron density. \citet{macneice_numerical_1984} provide two rules for adaptively adjusting the time step to ensure $f_k$ changes sufficiently slowly. Additionally, \citet{shen_lagrangian_2015} provide an alternate scheme for choosing the time step \textit{a priori} based on the input electron density and the eigenvalues of \autoref{eq:rate_mat}.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%                                   Appendix 2                                %
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Effective AIA Response Functions}\label{effective_response_functions}

% spell-checker: disable %
\begin{pycode}[manager_appendix]
from synthesizAR.instruments import InstrumentSDOAIA
from scipy.interpolate import splev
aia = InstrumentSDOAIA([0,1]*u.s, observer_coordinate=None)
em = EmissionModel.restore(os.path.join(manager_appendix.data_dir, 'base_emission_model.json'))
fig,axes = plt.subplots(
    2, 3, sharex=True, sharey=True,
    figsize=texfigure.figsize(
        pytex,
        scale=1,
        height_ratio=2/3,
        figure_width_context='columnwidth'
    )
)
T = np.logspace(5,8,100)
p = 10**(15)*u.K/(u.cm**3)
const_p_indices = np.array([(i,np.argmin(np.fabs(em.density.value-d.value)))
                            for i,d in enumerate(p/em.temperature)])
with h5py.File(os.path.join(manager_appendix.data_dir, 'effective_aia_response.h5'), 'r') as hf:
    colors = qualitative_palette(len([k for k in hf['94']])-1)
    for i, (ax, channel) in enumerate(zip(axes.flatten(), aia.channels)):
        grp = hf[channel['name']]
        real_response = splev(T, channel['temperature_response_spline'])
        ax.plot(T, real_response, ls='-',color='k',)
        ax.plot(em.temperature,
                np.array(grp['response'])[const_p_indices[:,0],const_p_indices[:,1]],
                color='k',ls='--',)
        elements = sorted([e for e in grp if e != 'response'],
                            key=lambda x: plasmapy.atomic.atomic_number(x))
        for j,element in enumerate(elements):
            ax.plot(em.temperature,
                    np.array(grp[element])[const_p_indices[:,0],const_p_indices[:,1]],
                    color=colors[j], ls='--', label=plasmapy.atomic.atomic_symbol(element),)
        if i==1:
            ax.legend(ncol=len(elements), loc="lower center", bbox_to_anchor=(0.5,1.02),
                      frameon=False)
        ax.text(6e7,1e-24,'{} $\mathrm{{\AA}}$'.format(channel['name']),
                fontsize=plt.rcParams['legend.fontsize'],
                horizontalalignment='right', verticalalignment='top')
ax.set_xscale('log')
ax.set_yscale('log')
ax.set_ylim([2e-30,2e-24])
ax.set_xlim([1e5,8e7])
ax.set_yticks([1e-29, 1e-27, 1e-25])
axes[1,0].set_ylabel(r'$K_c$ [DN cm$^5$ s$^{-1}$ pixel$^{-1}$]')
axes[1,0].set_xlabel(r'$T$ [K]')
plt.subplots_adjust(wspace=0.,hspace=0.)
fig_aia_response = manager_appendix.save_figure('aia-response')
fig_aia_response.caption = r'SSW temperature response functions (solid black) and effective temperature response functions for the elements in \autoref{tab:elements} (dashed black) for all six EUV AIA channels. The colored, dashed curves, as indicated in the legend, denote the contributions of the individual elements to the total response. For this calculation, we have assumed equilibrium ionization and a constant pressure of $10^{15}$ K cm$^{-3}$. We do not account for the time-varying degradation of the instrument.'
fig_aia_response.figure_env_name = 'figure*'
fig_aia_response.figure_width = r'\columnwidth'
fig_aia_response.placement = ''
fig_aia_response.fig_str = fig_str
\end{pycode}
\py[manager_appendix]|fig_aia_response|
% spell-checker: enable %

As discussed in \autoref{atomic}, the assumption of ionization equilibrium is likely to be violated in the impulsive heating cases considered here. Thus, we must recompute the contributions of each ion to the total channel response, using the result of \autoref{eq:nei_mat} in place of the equilibrium ion population fractions. \autoref{fig:aia-response} shows the effective temperature response functions for the six EUV channels on AIA compared to those calculated from \texttt{aia\_get\_response.pro} in SSW. Even though we include a limited number of transitions from the CHIANTI database (see \autoref{tab:elements}), we recover nearly all of the response from each channel. The high-temperature contributions in the SSW functions are due to continuum emission which we do not include in our model. In all cases, the continuum contribution is several orders of magnitude below peak of the channel response. Additionally, we do not account for the time variation in the wavelength response functions due to the degradation of the detector \citep[see Section 2.1.6 of ][]{boerner_initial_2012}.

% spell-checker: disable %
\begin{pycode}[manager_appendix]
fig = plt.figure(figsize=texfigure.figsize(
    pytex,
    scale=0.5,
    height_ratio=1.0,
    figure_width_context='columnwidth'
))
ax = fig.gca()
min_T = 1e300*u.K
max_T = 0*u.K
colors = heating_palette()
ar = synthesizAR.Field.restore(os.path.join(manager_appendix.data_dir, 'base_noaa1158'), lazy=True)
i_loop=680
loop = ar.loops[i_loop]
for i,h in enumerate(heating):
    loop.parameters_savefile = os.path.join(manager_appendix.data_dir, f'{h}', 'loop_parameters.h5')
    ax.plot(loop.electron_temperature[:,0], loop.density[:,0], color=colors[i], alpha=0.75,
            label=h.split('_')[0].capitalize())
    min_T = min(min_T, loop.electron_temperature.min())
    max_T = max(max_T, loop.electron_temperature.max())
p = 1e15*u.K/(u.cm**3)
T = np.linspace(min_T,max_T,1000)
ax.plot(T, p/T, color='k')
ax.set_xscale('log')
ax.set_yscale('log')
ax.set_xlim(1.75e5,1e7)
ax.set_ylim(1.1e7,6e9)
ax.legend(ncol=3,loc="lower center", bbox_to_anchor=(0.5, 1.02), frameon=False,)
ax.spines['top'].set_visible(False)
ax.spines['right'].set_visible(False)
ax.spines['bottom'].set_bounds(2e5,1e7)
ax.spines['left'].set_bounds(2e7,6e9)
ax.set_ylabel(r'$n$ [cm$^{-3}$]')
ax.set_xlabel(r'$T$ [K]')
fig_hydro_phase_space = manager_appendix.save_figure('hydro-phase-space')
fig_hydro_phase_space.caption = r'$n-T$ phase-space orbits for a single strand for the three heating scenarios as defined by the legend. The black line indicates a constant pressure of $10^{15}$ K cm$^{-3}$.'
fig_hydro_phase_space.figure_width = r'0.5\columnwidth'
fig_hydro_phase_space.fig_str = fig_str
\end{pycode}
\py[manager_appendix]|fig_hydro_phase_space|
% spell-checker: enable %

Furthermore, during the evolution of a strand, the pressure is not constant for any of our heating scenarios as evidenced by \autoref{fig:hydro-phase-space}. The black line of constant pressure $p=10^{15}$ K cm$^{-3}$ shows the default pressure at which the SSW AIA response functions are evaluated. The other lines show the temperature-density phase space evolution for the high-, intermediate-, and low-frequency cases for a single strand, none of which is well described by the assumption of constant pressure. By recomputing and interpolating the emissivity to the temperatures and densities as defined by our hydrodynamic simulation, we ensure that we are evaluating all quantities in \autoref{eq:ppuv} at the correct temperature and density.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%                                   Appendix 3                                %
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Computing Time Lags}\label{timelag_details}

To find the associated time lag for a channel pair in a given pixel, we compute the cross-correlation between the time series associated with each channel and find the delay which maximizes this cross-correlation. We can express the cross-correlation $\mathcal{C}$ between two channels $A$ and $B$ as,
\begin{equation}\label{eq:cc_pre}
    \mathcal{C}_{AB}(\tau) = \mathcal{I}_A(t)\star\mathcal{I}_B(t) = \mathcal{I}_A(-t)\ast\mathcal{I}_B(t)
\end{equation}
where $\star$ and $\ast$ represent the correlation and convolution operators, respectively, $\tau$ is the lag and
\begin{equation*}
    \mathcal{I}_c(t)=\frac{I_c(t)-\bar{I}_c}{\sigma_{c}},
\end{equation*}
is the intensity of channel $c$ as a function of time, $I(t)$, centered to zero mean and scaled to unit standard deviation. Taking the Fourier transform, $\mathscr{F}$, of both sides of \autoref{eq:cc_pre}, using the convolution theorem, and then taking the inverse Fourier transform, $\mathscr{F}^{-1}$, gives,
\begin{align}\label{eq:cc}
    \fourier{\mathcal{C}_{AB}(\tau)} &= \fourier{\mathcal{I}_A(-t)\ast\mathcal{I}_B(t)}, \nonumber\\
    &= \fourier{\mathcal{I}_A(-t)}\fourier{\mathcal{I}_B(t)}, \nonumber\\
    \mathcal{C}_{AB}(\tau) &= \inversefourier{\fourier{\mathcal{I}_A(-t)}\fourier{\mathcal{I}_B(t)}}.
\end{align}
Scaling \autoref{eq:cc} by the length of the intensity time series $I(t)$ yields the same result as that of the correlation defined in Section 2 of \citet{viall_evidence_2012}. Furthermore, the \textit{time lag} between channels $A$ and $B$ is defined as,
\begin{equation}\label{eq:timelag}
    \tau_{AB} = \argmax_{\tau}\,\mathcal{C}_{AB}(\tau).
\end{equation}

By exploiting the definition of the cross-correlation as given in \autoref{eq:cc}, we can leverage existing Fourier transform algorithms in order to compute $\mathcal{C}_{AB}$ in a scalable and efficient manner. For a 500-by-500 pixel active region observation and 15 possible channel pairs, we need to compute $\tau_{AB}$ nearly $4\times10^6$ times. We use the highly-optimized and thoroughly tested Fourier transform algorithms in the NumPy library for array computations \citep{oliphant_guide_2006} combined with the Dask library for parallel and distributed computing \citep{rocklin_dask_2015}. Using Dask, we are able to parallelize this operation over many cores such that, on a 64-core machine, we can compute time lags for all 15 channel pairs in every pixel of the \AR{} in less than ten minutes. For comparison, doing the same calculation by calling the IDL function \texttt{c\_correlate.pro} on each pixel in serial would take $\approx14$ hours for all 15 channel pairs.