Training Subjects On Key Concepts From Patient-Reported Outcomes (PROs) Improves Understanding And Data Accuracy

Introduction

The significance of patient-reported outcome data using electronic means (ePRO) is on the rise as the use of paper questionnaires has been shown to pose data quality and integrity issues.¹

Prior evidence has demonstrated that patients falsely inflate diary compliance. With a paper diary, reported compliance was 90%, but actual compliance was 11−20%.² Missing and inaccurate data contribute to a problem in clinical trials given data collected from subjects have a direct impact on the demonstration of drug efficacy.¹ Electronic diary compliance, on the other hand, was 94%,² and many studies have shown that patients actually prefer to use electronic methods to complete their diaries, leading to higher engagement than what has been seen with paper:

[...] smplified study design and schedule, collection of key assessments only, minimized burden on subjects and sites, and the combination of useful trial information into a single location, such as the eCOA trial device.³

are all key components to increase patient engagement and compliance.³ To further aid in compliance efforts, it is recommended that prior to the completion of the PRO/ePRO questionnaire, there should be education and training of subjects.⁴ Training includes education about the importance of compliance and the time period in which the diary should be used⁴ as well as a purpose statement about the assessment and best practices that confirm, for example, that the PRO questions should be answered as honestly as possible and that there are no incorrect answers.⁵ This educational information should be available to all subjects and conducted in a standardized environment to prevent any untoward bias on patient answers to the questionnaires.⁵

“Training and instructions to patients for self-administered PRO instruments” is listed in the FDA guidance on the use of PRO measures included for the purpose of clinical data quality control.⁵ More specifically, and due to the complexity of GI disorders, patient training has been highlighted by the FDA as highly recommended for the completion of the event log or diary along with complete instructions for the stool frequency and rectal bleeding assessments, and that this training can be accessed within the event log or diary.⁶ Further, recent FDA guidance for rare disease suggests, “a detailed description of procedures and training for performing the assessment may improve the reliability of the assessment.”⁷

Given the potential for improved compliance coupled with clear regulatory recommendations to implement training for studies with PRO related endpoints, this study aimed to examine subjects’ initial understanding of how to report on common patient-reported outcome (PRO) items and demonstrate the impact of a brief training on the accuracy of responses.

Methodology

Demographics

Between August and September 2017, 613 subjects across the United States completed a voluntary, anonymous online survey via an online resource (Clinical Connection). The survey was developed using Qualtrics (https://www.qualtrics.com) and distributed to individuals via email through clinicalconnection.com. Individuals were compensated for completion of the survey with entry into a drawing for a gift card.

Participants were included in the study if their primary language was English, and were adult male or female >=18 to 85 years old. Exclusion criteria included, non-English speaking <18 years old or >85 years old, and history of traumatic brain injury or severe neurological deficit that may make it difficult to complete an online survey. The mean age of all subjects was 46 years, in which 68% were female, 21% male, and 11% preferred not the answer. The majority of subjects attended some college (45%), 28% received a college degree, and the remaining completed a GED (23%) or no degree at all (4%). Subjects self-reported a broad range of diagnoses across various therapeutic areas, and 25% reported participation in a prior clinical trial. The survey questions presented here qualified for exempt status as determined by the Western IRB. Consent was assumed when each subject opted in to complete the anonymous, voluntary survey online.

Question And Training Development

First, a panel was developed consisting of scientists with expertise in the development and current use of PROs in clinical trials to develop key questions and conduct literature searches to identify evidence on which to develop recommendations for questions and responses. Literature searches were conducted for articles published from 2007 to 2017. All relevant identified English-language articles were included for review. No studies of sufficient quality were identified upon which to make evidence-based recommendations.

Next, we considered the content to be included in the training survey to assess insights into the patient understanding of terminology commonly seen in both standardized and homegrown PROs. The content was developed according to the following procedure: A panel of experts, as noted above, was invited to participate. Based on the literature, we identified the procedures, risks, benefits, and alternatives for the development of a survey to assess individuals’ perceptions and recommendations for questions related to the efficacy of subject training. Questions were chosen based on relevant therapeutic areas in which PROs are often used as primary or secondary endpoints, and in which there have been difficulties in patient compliance and PRO understanding. Training content was derived through the use of literature-based definitions and suggestions from the expert panel.

It was determined that situational questions would be used as they are similar to behavioral questions, but instead of asking the participant to relay a past experience and tell what they thought of it in that situation, they are presented with a hypothetical situation.

All participants were asked to complete a survey consisting of demographic questions (age, gender, highest level of education; current psychiatric or medical diagnosis) and situational questions followed by training. Subjects were first asked to answer a situational question to assess base knowledge. Next, subjects were presented with a small amount of information (1–2 sentences) on a scenario and subjects were re-asked the same questions. All questions kept a similar format presenting a scenario of themselves or a patient participating in a clinical trial and answering a question related to the indication. All subjects completed all questions. The correct answers were determined by consensus amongst the panel of scientists with expertise in PRO measures. Supporting rationale for the correct answers was based on literature review and past use of specific terminology in clinical trials.

Question 1: Pain

You are participating in a clinical trial that asks you to rate your pain by selecting the one number that best describes your pain at its worst in the last 24 hrs (0=no pain, 10=worst pain you can imagine). If your pain changed throughout the day (9 in the morning, 5 at noon, 1 in the evening when you’re completing your report), you should report …

Subjects were given the answer choices of 1, 5, or 9.

When reporting your worst pain on a daily basis, you should rate how severe your pain was at its worst point during the day. You should not average your pain over the day or report your pain level at the moment you are answering, but rather consider your pain level at its highest point that day.

Question 2: Rescue Laxative

Imagine you are participating in a clinical trial for constipation and are instructed to take rescue laxative as needed. What is a rescue laxative?

A rescue laxative is a medicine used to stimulate or facilitate a bowel movement when you are experiencing constipation. A rescue laxative is not the investigational medication that is being studied in the clinical trial.

Question 3: Stool Counts

If you were participating in a clinical trial that asked you to record the number of stools you had each day and you had a stool and returned to the bathroom 5 mins later and had a second stool, would you record this as 1 stool or 2 stools?

If you get off the toilet and back on again, no matter how much time elapsed in between or how much stool was passed from the first to second instance, you should record it as 2 stools.

Question 4: Bleeding Episodes

If you were participating in a clinical trial that asked you to report bleeding episodes into a diary, and you notice two bleeds that occur at the same time in different locations, how many new bleed entries should you enter into the diary?

You should report every bleed that occurs, one entry for each location, even if they occur at the same time.

Question 5: Itch Severity

Claire suffers from a chronic skin condition called psoriasis, for which she is participating in a clinical trial. Most of the time, Claire’s psoriasis-related itching is well-controlled, but when she is under a lot of stress, her psoriasis often flares and her skin is very itchy. If Claire was having an itch-free week but today she is under a lot of stress and her psoriasis flares, what should Claire report as her worst itch in the last 24 hrs in her study diary?

When reporting the severity of psoriasis-related itching on a daily basis, you should rate how severe your itching was at the worst point during the day. You should not average the itch, or rate itchiness level at the time of reporting on the itch.

Results

Question 1: Pain

Out of 613 subjects, prior to training, 78% of subjects (n=481) selected the correct answer; 461 subjects reported the correct response before and after training. After training, of the 132 subjects who first chose an incorrect response, 87 switched their answer to the correct response (McNemar’s Test, p < 0.0001), increasing post-training accuracy to 89% (Table 1).

Table 1 Response Summary To 5 PRO Questions

Question 2: Rescue Laxative

Out of 613 subjects, prior to training, only 24% of subjects (n=150) correctly understood the meaning of a rescue laxative; 127 subjects reported the correct response before and after training. After training, of the 463 subjects who first chose an incorrect response, 216 switched their answer to the correct response (McNemar’s Test, p < 0.0001), increasing post-training accuracy to 56% (Table 1).

Question 3: Stool Counts

Out of 613 subjects, prior to training, only 67% of subjects (n=408) selected the correct answer when first asked to count the number of stools within the scenario they were given; 368 subjects reported the correct response before and after training. After training, of the 205 subjects who first chose the incorrect response, 177 switched their answer to the correct response (McNemar’s Test, p < 0.0001), increasing post-training accuracy to 89% (Table 1).

Question 4: Bleeding Episodes

Out of 613 subjects, prior to training, only 85% of subjects (n=518) chose the correct answer when asked how many new bleed entries they should enter into a diary when participating in a clinical trial; 499 subjects reported the correct response before and after training. After training, of the 95 subjects who previously selected the incorrect response, 67 switched their answer to the correct response (McNemar’s Test, p < 0.0001), increasing post-training accuracy to 92% (Table 1).

Question 5: Itch Severity

A scenario and brief background is given to subjects regarding a patient who is participating in a clinical trial for psoriasis. Out of 613 subjects, prior to training, only 76% of subjects (n=466) chose the correct answer when asked what the patient should report as her worst itch within the last 24 hrs in her study diary; 437 subjects reported the correct response before and after training. After training, of the 147 subjects who chose the incorrect response, 67 switched their answer to the correct response (McNemar’s Test, p < 0.0001), increasing post-training accuracy to 82% (Table 1).

Collectively, for all 5 questions, the mean response accuracy significantly increased from 66 ± 21% to 82 ± 23% (t₆₁₂ = 18.017.9, p < 0.0001, Cohen’s d = 0.72; Figure 1) from pre- to post-training, demonstrating improvement in understanding of the terminology.

Figure 1 Mean accuracy response (5 PROs).

Discussion And Conclusions

Patient-reported outcomes (PRO) are inherently subjective. These measures are intended to capture the unique experience of each individual being assessed. One problem with reliable PRO data is inconsistency in the way subjects conceptualize the symptoms being measured. The interpretation of a specific clinical event may vary from subject to subject, and the same clinical event can be assigned different severity ratings within and across subjects. These varying interpretations of a patients’ own health status as reported in a PRO can be detrimental to primary endpoint data in a clinical trial, thereby increasing signal to noise and distorting the ability to assess true drug efficacy.

This study supports the evidence that even 1–2 sentences of simple educational instruction significantly improve patient understanding of the tested concept. Therefore, if a more comprehensive training was to be provided, such as a 5–10 min interactive course about the daily diary explaining definitions and key concepts, it would likely considerably increase patient understanding of the PRO measure.

Further, training could focus subjects on the expectations of the trial and their important role in the study, ultimately increasing diary compliance. By providing a training available for patients to review offline at any time, clinical trials would better align with global regulatory recommendations and collect more focused and clinically accurate endpoint data about patients’ health status, ultimately improving trial efficacy.

Limitations

In this study, the methods included a repeated measure within-subject design, in which we did not have a control group being asked the same two questions in succession with either no training or a different training. Future studies will address this limitation to determine and mitigate the impact of a potential learning effect on the data.