Supporting Psychometric Instrument Usage Through the POEM Ontology

Ontologies provide a formally structured way to represent knowledge by defining a shared vocabulary and the relationships between concepts within a particular domain [23] [32]. They enable both humans and machines to consistently interpret and reason over data. Ontologies are often expressed using standardized formats like the Web Ontology Language (OWL) [2], and are commonly used to integrate heterogenous data sources, support interoperability, and enable automated reasoning. Ontologies include definitions of basic concepts in the domain as a finite set of unambiguously identifiable classes and relationships [28]. While taxonomies and ontologies should be distinguished, a very lightweight ontology might also be classified as a taxonomy, and many ontologies contain hierarchies of classes that are based on existing taxonomies of concepts [44].

A number of widely used, comprehensive ontologies and taxonomies exist to give structure to psychological concepts such as diseases, syndromes, signs, symptoms, and assessments; some target the domains of psychology or neuroscience specifically, while others contain neuropsychological concepts within a wider medical context. These ontologies do not represent the internal structure of instruments and questionnaires, but in some cases model the constructs that certain tasks measure. The Systematized Nomenclature of Medicine (SNOMED) is the largest and most widely-used of these; it was created in 1966 as a logic-based healthcare terminology, and has evolved into its current form, SNOMED Clinical Terms (SNOMED CT), which is the most comprehensive healthcare terminology internationally [11]. SNOMED CT contains many psychological concepts, including those subsumed under clinical findings, disorders, and attributes, and acts as an interoperability standard for healthcare information exchange in the United States. Although it is primarily a polyhierarchical taxonomy, SNOMED CT has an ontological foundation and is freely available as an ontology. SNOMED CT can be utilized to handle queries over medical systems in which it is used, due to its support of a simple description logic.

The structure of psychiatric conditions has been formalized for clinical purposes, the most prominent iterations of which are the International Classification of Diseases (ICD), currently on its 11th revision, and the Diagnostic and Statistical Manual of Mental Disorders (DSM), 5th revision [1]. The ICD-11 [33] is a medical taxonomy published by the World Health Organization (WHO); its content encompasses a wide range of health conditions, with a chapter regarding mental, behavioral, and neurodevelopmental disorders. This standard is used internationally, for statistical comparison of morbidity as well as clinical purposes. Currently mappings from ICD-11 codes to the equivalent SNOMED CT concepts are not available, presumably because the ICD-11 is a recent revision and not completely integrated in all settings, but such a map for the ICD-10 is made available by SNOMED International [21]. The DSM-5 [1] is similar in taxonomy to the ICD with key differences: the DSM contains psychiatric diagnoses only, and is based on mental health data and standards from the United States only, limiting its applicability and reach, but it is commonly used in the US for psychiatric diagnoses and treatment recommendations. Although neither is formalized as an ontology, each is an important source of structured mental health information to consider due to their ubiquity in clinical and research settings.

The Cognitive Atlas (CogAt) [36] is a knowledge base for cognitive neuroscience that aims to map mental processes onto brain function, while addressing issues with ambiguous terminology in the field. Additionally, CogAt adopts Wikipedia’s approach to collaborative knowledge building, capturing disagreement by allowing public contributions and discussion. CogAt uses basic ontological relations as included in the OBO Relational Ontology, as well as relations between processes, tasks, and “descended from” relations between tasks. The basic classes subsuming all entities in CogAt are Concepts, Tasks, Phenotypes, and specialized Collections of tasks and theories. Although CogAt’s hierarchy of Disorders is derived from the Disease Ontology (DO) database [40] and matches most other standard nosologies, CogAt does not aim to be as comprehensive in its inclusion of symptoms.

There are several existing standards for data exchange in clinical settings. For example, Logical Observation Identifiers Names and Codes (LOINC) [18] provides a common terminology for laboratory and clinical observations for clinical care and management. It has a rich catalog of clinical documents and standardized survey instruments, included psychometric assessments, and is available for public use. Similarly, the Health Level 7 (HL7) clinical document architecture contains many different standards supporting the interoperability of medical data. One such standard is Fast Healthcare Interoperability Resources (FHIR), which provides detailed specifications for document structure including for questionnaires [14].

Several ontologies exist to capture the semantics of measurement instruments; here, we consider those that are close to the goals of POEM either in domain (measurement of constructs in behavioral science, psychology), or in the depth of semantics covered. Beginning in 2010, Batrancourt et al. [3] [4] developed an ontology of Mental State Assessment (ONL-MSA) as an extension of the OntoNeuroLOG ontology [43], a common semantic model of neuroimaging data and tools. ONL-MSA’s fundamental contribution is a core ontology providing a general model of mental state assessments alongside a taxonomy of behavioral, neuropsychological, and neuroclinical instrument types. Instrument properties captured include decomposition into sub-instruments, definitions of associated variables, and the domains and qualities measured by instruments. Instruments are associated with a set of measuring actions that link resulting scores to measured variables, and numeric scoring scales are expressed with a set of codes.

In 2012, Cox et al. developed a pair of complementary ontologies to extend the Ontology for General Medical Sciences (OGMS): the Neurological Disease ontology (ND), which aims to be a comprehensive representation of all facets of neuropsychology, and the NeuroPyschological Testing ontology (NPT) [12] [13], a set of classes that represent cognitive functioning assays, including tests, cognitive functions measured, scores, and the associated scale results. NPT and ND also utilize the Mental Disease (MD) and Mental Function (MF) ontologies created by Hastings et al. [24]. Primarily, NPT aims to annotate neuropsychological tests and testing data in order to integrate results over a range of assessments in overlapping domains.

Bensmann et al. have created an ontology with the aims of facilitating data search and reuse for social science survey items [7]. Social surveys, which comprise items usually about attitudes, behaviors, and factual information, are currently findable based on survey-level metadata, but question-level and variable-level searches have previously not been available. Bensmann’s ontology tries to remedy this through annotation of question features including the nature of the problem or task given to the respondent, the tone and complexity of language used, and the nature of objects in question, among others.

We have generated a set of use cases, along with competency questions that would be encountered in a broad range of interactions with psychometric assessments, accompanied by example answers based on the full 47-item Revised Child Anxiety and Depression Scale (RCADS-47) [10] and the shortened 25-item version (RCADS-25) [16]. These were generated through regular working sessions that included scenario walkthroughs and expert review, informed by the context of pragmatically supporting RCADS user support service, beginning with questions most frequently received as a starting point. Potential use cases identified are as follows:

• $\mathrm{■}$

  Clinical service: assessment – finding the most appropriate assessment for a specific patient and context based on conditions measured, available languages and norms, provenance, and metrics showing primary utility and exposing strength of evidence; elucidating instrument usage instructions, target subject and respondent, and what an instrument measures

• $\mathrm{■}$

  Clinical service: monitoring – finding the most appropriate assessment for a specific patient and context, which may include reuse of initial questionnaires and scales, or establishing confidence in a shorter assessment for ease of repeated use; elucidating instrument usage instructions, target subject and respondent, and what an instrument measures

• $\mathrm{■}$

  Research: production – determining whether appropriate measures already exist, identifying where reliability and validity may be improved by iterating on an assessment or creating a new one, and supporting the semantic representation of new measures

• $\mathrm{■}$

  Research: synthesis – supporting the process of combining the results of multiple studies by providing standardized representation of both external and internal questionnaire structure, metadata, and variables

• $\mathrm{■}$

  Development: translation and derivation – determining whether specific language translations of questionnaires exist, as well as their reliability and validity (which may be very different from the original language iteration), determining the minimum design elements needed to create a valid derivative measure, and directing developers towards the provenance and meanings of constructs measured, which should support the accurate translation or adaptation of questionnaires in both linguistic and cultural contexts

Questionnaire-level competency questions can be seen in Table 1, with indication of the particular use cases they are each useful to. Competency questions relevant to specific questionnaire items and scales are relevant across use cases, and can be seen in Table 2.

Based on our competency questions and continual dialogue between the clinical psychology researchers and semantic researchers on our team, we have established a terminology list containing the concepts and definitions that have emerged as important, maintained concurrently with a UML diagram (Figure 1) representing the structure of POEM as well as its alignment with the foundational ontologies we have chosen to use. Additionally, we developed prototype applications that exposed whether the entities and relationships modeled were sufficient to enable tools serviced by user-relevant data structures, continually iterating between POEM revisions and the design of these applications; particularly, tools associated with the RCADS website for accessing information on the RCADS assessment instruments.

The main goal of POEM is to provide a logical and systematic description of measures in clinical psychology. Based on this goal, as well as terminology and use case documentation, we constructed the POEM ontology focusing first on basic questionnaire structure, then underlying constructs, and finally, evidence modeling, with continuous iteration based on feedback from domain experts. Throughout this process we have used the RCADS-25 and RCADS-47 as examples with which to evaluate our progression, while ensuring that the structure defined by the terms and relations in POEM generalize to other measures in clinical psychology.

POEM is implemented in OWL/RDF [29], with the help of the open-source ontology editor Protégé [30]. We maintain POEM using GitHub at https://github.com/tetherless-world/POEM, and documentation including use case deliverables, terminology, a static demo, and publications at https://tetherless-world.github.io/POEM/.

POEM utilizes several well-established ontologies at different domain levels in order to effectively build off of previous work and provide inherent modularity in its usage. Prefixes used for modules of the ontology are shown in Table 3.

The core concept of POEM is the psychometric questionnaire, which inherits the majority of its structural concepts from questionnaire (vstoi:Questionnaire). The psychometric questionnaire concept inherits provenance attributes such as authorship, licensing, intended subject and respondent, derivation, and structural attributes such as instructions. Additionally, the questionnaire item (vstoi:Item) encapsulates several entities and other pieces of information included in an assessment question: an item stem, which is the text presented in the context of prompting a response regarding some latent construct; the item stem concept, which is the specific phenomenon an item is intended to capture; and a codebook (vstoi:Codebook) representing a range of possible respondent experiences. Items have instrument membership attributes inherited from HAScO specification, giving items membership and position within any number of questionnaires.

The primary focus of POEM lies in the representation of the underlying semantics of psychometric questionnaires. This modeling extends from the foundational structure established in VSTO-I, representing the constructs (poem:Construct) that questionnaires assess. A psychometric construct is a phenomenon whose signal is meant to be detected by an item or questionnaire scale.

An item is an indicator that targets a signal representing a particular construct, using the recorded human response. More precisely, an item stem is associated with an item stem concept, which is an expression of the specific construct or experience intended to be captured. Further, questionnaire scales (poem:QuestionnaireScale) comprise a collection of one or more item stem concepts designed to measure a shared construct. The set of item stem concepts in a scale that has been shown as reliable and valid is said to accurately estimate the presence of a construct.

To support a consistent view of variables for data unification, HAScO uses the notion of a semantic variable[35]. Formalized variable specifications support the alignment and combination of variables in processes that occur across multiple studies. In particular, the semantic variable includes entity and attribute properties, but no population property. Accordingly, two variables share a common semantic variable when the only distinction between them is their population properties. HAScO specifies that semantic variables are measured by detectors, and are also associated with attributes such as temporal and spatial information, and measurement unit. In POEM, the semantic variable detected by an item has, for its attribute, some symptom, and for its entity, a human of some demographic. In this way, questionnaire data can be aligned within and across cohorts.

POEM introduces the experience class. Each codebook corresponds with an experience related to the subject’s personal degree to which they experience a construct measured by a questionnaire item; for example, frequency or intensity.

We maintain that POEM should remain neutral with respect to frameworks such as the ICD or DSM, and recognize that these and related frameworks are not uncontroversial. Further, while POEM provides the scaffolding for linking questionnaire scales to clinical constructs, it does not limit these constructs to those in a particular nosology. The axiom <poem:QuestionnaireScale isAbout poem:Construct> is intentionally generalizable, allowing integration with any framework, provided that any object of this axiom is reasonable subsumed by the construct concept. In our evaluation, we use SNOMED entities to represent the concepts measured, because of its wide use and coverage, and mappings to other terminologies.

Determining if psychometric tests are adequate to assess psychological constructs is a process that is crucial to successful clinical assessment. POEM supports these processes by modeling population-based studies (SIO:001041) that are conducted by researchers to determine whether features of questionnaires measure the construct they are intended to, including metrics such as reliability and validity. The study group population that meets some criteria for inclusion in a study is referred to as a cohort (hasco:cohort). Knowing the context in which a measure is applied can be important to evaluating its importance. We use SCO, which connects the cohort class to demographic information, effectively connecting studies to the context in which they were conducted.

Observational studies involve the administration of created measures to a cohort one or more times depending on the target metric; for example, measuring test-retest reliability requires comparison between scores within the same cohort over time. Questionnaires such as the Caregiver version of the RCADS-47 require the inclusion of two cohorts, since the respondent (caregiver) and subject (youth) are different people. Each metric produced as the result of a study constitutes a piece of evidence (ECO:Evidence), which supports some assertion about an item or scale of the instrument in question. For example, in an initial study of the RCADS-47 [10], the value of Cronbach’s Alpha ($\alpha$) calculated for the Social Phobia scale was $\alpha =0.82$. This value of $\alpha$ is generally considered to be good; so, the Cronbach’s Alpha coefficient for the Social Phobia scale is a piece of observational study evidence supporting the assertion that the Social Phobia Scale has good internal consistency for the cohort represented. The modeling described can be seen in Figure 2. This formalization of studies and evidence supports the aggregation of metrics about a questionnaire and its features, allowing clinicians to choose questionnaires that adequately fit their use case.

We avoid making claims about what standards constitute “good evidence”; for example, while a value of $\alpha =0.82$ is generally considered good, there is no consistent criteria for determining assessment quality, and there are certain ways in which numerical indicators can be misleading [38].

We demonstrate POEM’s utility in psychometrics via the competency questions posed early in the development process (table 1, table 2), using the RCADS-47 and RCADS-25 as subjects. We revisit the use cases outlined in Section 3.1 along with applicable competency questions, based on knowledge graphs representing the 47- and 25-item versions of the RCADS. For each competency question, we deploy SPARQL to query the complete knowledge graphs.

Table 5 revisits the competency questions posed in tables 1 and 2, along with specific answers based on the RCADS-47 and RCADS-25. Table 5 also shows how the current version of POEM can support the answering of a subset of our original competency questions through SPARQL queries, generating results that align with SME-generated answers. The SMEs reviewed the ontology’s structure and its ability to provide meaningful responses, through demonstration of queries that could sufficiently answer the generated competency questions dynamically, provided by the ontology experts. Overall, SMEs expressed a high degree of satisfaction with the ontology’s coverage and utility.