Tactical Considerations for Designing Real-World Studies: Fit-for-Purpose Designs That Bridge Research and Practice

Introduction

Real-world evidence (RWE) that can help guide clinical practice is derived from real-world data (RWD), defined by the US Food and Drug Administration (FDA) as “data relating to patient health status and/or the delivery of health care routinely collected from a variety of sources.” The FDA defines RWE as “clinical evidence about the usage and potential benefits or risks of a medical product derived from analysis of RWD.”¹ Simply put, RWD is the collective term generally used to describe data collected outside of a traditional randomized clinical trial (RCT) research setting, often in patient-clinician encounters and/or from information generated by patients in terms of their perception of pain, function, quality of life, and so on, and/or from biosensors worn by study participants.²

Randomization may be introduced at baseline to facilitate balanced comparison groups for pragmatic clinical trials (PCT), which then assume many of the characteristics of RWE after randomization, such as studying treatments as used rather than according to intent-to-treat analyses and including more diverse populations.^3,4 PCT generally focus on health outcomes that inform a clinical or policy decision,^5,6 in contrast to intermediate and surrogate outcomes often studied in traditional RCT prepared for regulatory approval of new medical products and supplementary new indications, which may be less reflective of true clinical outcomes.⁷

RWE is becoming widely recognized as a complement to results from RCTs, especially as we recognize the importance of learning about how diverse patients respond to treatments administered outside of the clinical trial setting and the durability of any such benefits, especially for vulnerable patients who are often not included or underrepresented in traditional RCTs.⁸ It is important to emphasize that RWE is not a replacement for traditional RCTs but rather an important supplement used to fill evidence gaps remaining after a product has received regulatory market authorization.^9,10 The evidence provided by RWE can inform medical practitioners about treatment effectiveness, safety and heterogeneity of treatment response, and can often be conducted more quickly than a traditional RCT and at lower cost. RWE is frequently the choice for studying the natural history of disease, evaluating care pathways,¹¹ developing clinical treatment guidelines and assessing their implementation,¹² and providing context for medical product marketing authorizations and label expansions in the US, Europe, and major markets in Asia.^13–17

In real-world settings, patients present with clinical histories and risk factors that may be substantially different from those studied in most RCTs. Tan et al¹⁸ recently reviewed all the RCTs listed on ClinicalTrials.gov and other public trial registration sites, recorded the indications under study and their inclusion and exclusion criteria (I/E). When these trial I/E exclusion criteria were applied to electronic medical record data from the UK National Health Service, more than 50% of patients with relatively prevalent and/or costly conditions would not have been eligible for these RCTs due to their age, co-morbidities, and/or use of prescription medication(s) for unrelated conditions. This demonstrates the impetus for the research community to use RWE to fill the evidence gaps about more diverse patients.

By way of perspective, RWE is currently being used by regulators as context for single-arm studies, largely in oncology and rare diseases – situations where randomization may not be feasible or ethical.^9,19 Blinatumomab, for example, was approved for relapsing refractory acute lymphoblastic leukemia based on promising data from a Phase 2 trial.²⁰ The market authorization holder created an external control arm from pooled historical clinical data to support the interpretation of their single-arm trial and secured a label expansion before the Phase 3 RCT was completed.²¹ Similarly, both the US FDA and the European Medicines Agency approved cerliponase alfa to slow the loss of ambulation based on a comparison of 23 treated symptomatic patients and 42 historical controls drawn from RWD.²²

RWE has also been used to assess learning in risk management programs and to monitor adherence to guidelines, such as the “Get With The Guidelines” studies in coronary artery disease and heart failure.²³ Looking forward, machine learning models use RWD to understand disease progression, for example, by identifying rheumatoid arthritis patients who will switch from methotrexate to a biologic 30 days prior to occurrence of the medication change.²⁴

Here, we describe four approaches used to generate RWE and provide pointers for clinicians interested in study design and execution. We offer tactical guidance about (1) opportunistic study designs, (2) considerations about representativeness of patients selected for study, (3) the need for transparency of data provenance, data handling, and data quality assessments, and (4) considerations for strengthening studies using record linkage and randomization in pragmatic clinical trials.

Study Designs for Real-World Data

Study design is foundational to developing reliable RWE. Much of the RWE that informs clinical practice comes from four broad types of study designs: (1) studies derived from patient registries, (2) follow-up after participation in traditional RCTs (sometimes known as “roll-over” studies) and other long-term safety studies, (3) pragmatic randomized trials, and (4) evidence hubs populated with RWD.

• Patient registries are used for many purposes including studying the natural history of disease, product safety, characterization of high-risk patients and identification of unmet needs. A strong advantage of patient registries is their adaptability. Broad data collection can be used to support a number of studies, and data collection may be adapted over time to include new exposures and outcomes of interest. See, for example, the registry maintained by the Cystic Fibrosis Foundation at www.cff.org. Registries have also been used to study physician decision-making, eg, the impact of PET and PET/CT scan results on intended management of various cancers.²⁵
• Roll-over studies and long-term safety studies are follow-up studies designed to evaluate the durability of benefits and long-term safety, including the risks and benefits of various treatment sequences and combinations. In fact, post-market authorization long-term cohort (follow-up) studies are attracting renewed attention due to regulatory requirements to follow patients who receive cell and gene therapies for 5–15 years, as well requiring long-term studies for other treatments with limited follow-up data available at launch and/or for medical products approved using surrogate outcomes.
• Pragmatic randomized trials use baseline treatment randomization to achieve balanced comparison groups with regard to unmeasured confounders and then use naturalistic follow-up. These studies can also be used to understand surgical methods and patient management, not just treatments.
• Evidence hubs use multiple linked RWD sources, with data review and management (data curation) conducted in parallel with data collection. They are similar in concept and in benefit to RCT protocol approaches such as basket and umbrella trials and can be used to address questions as they arise, while also continuing to gather new data.²⁶

Each study starts by crafting one or more research questions, identifying the target study population, exposures/treatments, and outcomes of interest. The next step is to determine whether most or all necessary data elements are available in existing records like electronic health data or health insurance claims, or if additional data collection is needed. A variety of guidance documents are available to assist with study design and evaluation. User Guides for Patient Registries²⁷ and for Developing Protocols for Observational Comparative Effectiveness Research²⁸ provide detailed information to help with design and execution.

Feasibility and data quality assessments are critical, both at study conception and during design as needed. An emphasis on assessing data availability and completeness, alongside estimation of the size and distribution of a potential pool of study subjects, is key to understanding whether a research question can be reliably answered. That said, data completeness is often misunderstood by a lack of recognition that test data, for example, may appear to be missing because the test was not ordered and not because the test value was not recorded. While this information is still missing, it should not be attributed to faulty data quality but instead reflects the actual practice of clinical care. Also important, since many RWE studies are exploratory in nature, statistical considerations about multiple comparisons and reduction of alpha are not problematic; the goal of these studies is to estimate effect sizes, not test hypotheses.²⁹

Tactical Guidance

In the spirit of pragmatism, we offer the following high-level guidance for study design:

Keep a Broad Eye Open to Naturalistic Opportunities for Study Design

Is there a natural experiment you could construct using existing data that would inform clinical decision-making by providing needed evidence? For example, hypersensitivity reactions following intravenous high-dose iron infusions with ferric carboxymaltose or isomaltoside 1000 were compared following switches in departmental purchasing over time.³⁰ Another naturalistic experiment was created using RWD from a large health-care organization in Israel to evaluate the safety of messenger RNA-based vaccines against severe COVID-19, a study that has been described as “the most reliable scientific tool” the world had to evaluate the effectiveness of the [Pfizer] COVID vaccine and the impact of the vaccination program.^31,32

The value of person-generated health data (PGHD) has also come to light for learning about people’s perceptions of vaccine safety. PGHD, also known as patient-reported data, refers to health-related data created, recorded, and/or gathered from patients, family members, and/or caregivers without influence or intervention of clinical staff to help address a health concern.³³ During the pandemic, community-based volunteers were recruited online and asked if they had been vaccinated and if so, with which vaccine; if they noticed any side effects following vaccination; had sought medical attention for any such side effects or had tested positive for COVID-19 since vaccination. Follow-up over 30 days showed similar effectiveness and side effects for the three vaccines approved in the US at that time.^34–36

Do Not Sacrifice Relevance for Representativeness

While studies are often criticized for not being geographically or demographically representative, this criticism does not hold up to scrutiny.³⁷ A study need not be broadly representative to be useful. Well-described study populations can provide important information unique to their patient characteristics, eg, race, ethnicity, and care settings.

Consider how large, closed-cohort occupational health programs for professional athletes were used to further understanding about COVID-19 transmission,³⁸ the effectiveness of COVID-19 diagnostic tests approved under Emergency Use Authorizations,^39,40 the impact of post-recovery viral shedding on test results and transmission,⁴¹ and the ability of vaccine boosters to prevent incident infection.⁴² This RWE was useful due to its timeliness, relevance, and on-going quality controls conducted as part of this evidence hub, including linked data from wearable devices, daily diagnostic results, contact tracing, and infection/vaccine history. Although these cohorts were largely composed of relatively healthy males, the evidence had broad relevance, with likely applicability to women, youth, elderly, and non-athletes.

Data Quality Matters. Documentation of Data Management, Staff Training, and an Audit Trail are Expected

It is important to understand why, where and how RWD were created and whether the data is likely to have been accurately and consistently recorded.⁴³ A clear description of the study population and data provenance, characterized in terms of person, place, and time, along with a description of data management and any data transformations will be expected.⁴⁴ The goal here is to provide enough information for reviewers to understand how the study population was recruited and the data were collected, handled, and analyzed, both to assist in interpretation and also so that others may replicate their methods, to the extent feasible, in different populations to evaluate whether the findings are broadly generalizable.

Most real-world studies use little, if any, source data verification. Instead, data are reviewed and curated, ideally on an ongoing basis rather than waiting until the end of the study. Established coding systems and algorithms should be used; exceptions should be justified. It is worth noting that roll-over studies and other safety studies may use MedDRA coding (Medical Dictionary for Regulatory Activities), which requires a potentially challenging mapping to ICD coding (International Classification of Diseases) used to classify disease, injuries, and other health conditions, as developed by the World Health Organization.

Purpose-driven data curation should focus on key variables and correction of data when possible and as needed. Supplementary data quality checks and/or validation for outcomes can be useful. For example, in creating the National Football League (NFL) Injury Analytics program, media reports of player injuries were compared with those reported through the NFL electronic medical record system to evaluate and enhance completeness of reporting, noting that injury descriptions from medical staff, not the media, were always considered the gold standard.⁴⁵

There are a number of guidelines and frameworks available to guide quality review and data curation. For example, the Kahn Framework harmonizes a number of established data quality frameworks to define a comprehensive assessment method for the quality of electronic health record data used in secondary settings.⁴⁶ This conceptual framework assesses quality of secondary EHR data across three dimensions of conformance, completeness, and plausibility, each of which is assessed through both the context of verification and the context of validation. Data quality frameworks such as the Kahn Framework are typically paired with standardized data structures, such as common data models (CDMs), to enable the standardization of collection, transformation, and review of data. Common data models can be designed with an eye toward data-types, such as the Observational Medical Outcomes Partnership (OMOP)⁴⁷ CDM are designed as a standard structure for capture of observational data, or they can be designed to standardize the capture of data unique to specific therapeutic areas, such as the mCODE data standard,⁴⁸ which defines the minimum common data elements needed for collection for oncology research.

The greater the potential impact of a decision, eg, approving a new indication for a marketed product, the more validations and data checks will be required.⁴⁹ If study results are intended for use in a regulatory submission, such as to support data on lack of effect in a non-treated group or natural history studies,⁵⁰ a detailed audit trail will be expected to ensure transparency in data provenance. Safety reporting is mandatory when funding is provided by a company that holds a market authorization for a product under study.⁵¹

Consider Strengthening a Study Using RWD Linkage, Patient-Generated Health Data Including Digital Health Technologies, and/or Treatment Randomization

Will new data be needed and from what sources? Long-term safety studies and roll-over studies often collect data directly from patients or their caregivers, only seeking clinical confirmation for events of special interest. The possibility of linking patient data to other medical information should be considered. Record linkage can help with long-term follow-up, historical patient information, clinical confirmation, or overall augmentation of foundational health records or registry data. The types of RWD that are often used for linkage include administrative health insurance claims data, pharmacy claims, and regional or national death records.^52–54 In a first-ever FDA acceptance of RWE as substantial evidence (in contrast to supporting evidence), transplant registry data were linked to US Social Security Administration death master files to support a label expansion for tacrolimus to prevent organ rejection in patients receiving lung transplantation.¹³

Other RWD can also be linked, and studies based on these linked data illustrate the utility of PGHD, genomic data, laboratory results, and direct-to-patient mobile data as part of an evidence hub. This approach is useful across clinical and occupational health settings. For example, some elite sports organizations were early adopters of injury surveillance programs linked with other RWD to improve athlete health.^45,55 These evidence hubs integrate information from electronic medical records (EMR) with player participation, game statistics, sideline clinician reports, equipment data, and wearable devices, and utilize regular quality reviews to support the reliability of ongoing research findings.^56,57 During the COVID-19 pandemic, these hubs were recrafted to provide a clinical foundation and longitudinal diagnostic data from frequent COVID-19 surveillance testing within a closed cohort,^38–40 serological testing for SARS-CoV-2 antibodies,⁴¹ and genomic sequencing performed for all infections to determine the SARS-CoV-2 variant and vaccination history.⁴² These RWD formed the basis for timely reliable evidence, including demonstrating that recovered individuals who continued to test positive for SARS-CoV-2 following discontinuation of isolation were not infectious to others,⁴¹ understanding the viral trajectory of illness,^58,59 and demonstrating that booster vaccinations were associated with a significant reduction in incident infections during the Omicron wave.⁴² Studies of the NFL occupational cohorts used data from wearables worn by players and staff while in their facilities and during travel to drive insights on disease transmission, showing that COVID-19 could be transmitted with less than 15 min of contact³⁸ and that there was great variability in the accuracy of testing from newly approved diagnostics.³⁹

Treatment randomizations may be useful in helping avoid key sources of bias, such as selection bias and imbalance in baseline risk factors, by randomly allocating patients to treatment or control, a technique that lends itself to use in registries and health systems or as a stand-alone approach.⁶⁰ The TASTE (Thrombus Aspiration in ST-Elevation Myocardial Infarction) randomized registry trial, for example, randomized patients from an existing registry, substantially lowering the cost of patient recruitment and speeding time to completion since the data of interest were already being collected as part of the registry protocol.⁶¹ Similarly, the Diuretic Comparison Project by the US Veterans Administration is a switching study where participants who were using hydrochlorothiazide diuretics, 25 or 50 mg daily, were randomized at the point of care (here, the pharmacy) to either stay on their current regimen or switch to chlorthalidone at a suggested equipotent dose. Patients enroll in the study with permission from their primary care provider and informed consent is obtained by telephone.⁶²

Randomization adds confounding control and simplifies the interpretation of study findings, which can be particularly useful to differentiate treatments in a crowded market. By way of example, the PRIDE (Palmitate Research in Demonstrating Effectiveness) study was conducted to study real-world outcomes in schizophrenics who had recently been incarcerated, hospitalized for an episode of psychosis, attempted suicide, and/or engaged in high-risk behaviors.⁶³ This study group was chosen since it is easier to detect a true effect (benefit or harm), should one exist, in a group at high risk for the events of interest. Participants were followed in a time-to-relapse study after being randomized to a monthly injection of paliperidone palmitate or their choice of one of seven daily oral medications. This practical comparison yielded RWE describing the comparative effectiveness, which was then included in the Patient Exposure section of the label (section 6.1).

Discussion

Clinicians are the bedrock of data creation and data validation. Community settings are integral to understanding treatment heterogeneity among diverse populations, how marketed medical products are used in everyday life, and which patients are most likely to benefit or be harmed and under what circumstances. The systematic use of RWE will not only contribute to evidence-based clinical practice and decisions by regulators and payors around product approval, effectiveness, and safety,⁶⁴ but may also facilitate clinician-led discovery of off-label drug therapies.⁶⁵

Expertise specific to the design and conduct of studies that use RWD cannot be overlooked, as these methods are complex. High-quality studies that use RWD generally start with feasibility assessments to assure that critical exposure(s), outcome(s) and covariates are recorded or able to be collected⁶⁶ and require study designs that are appropriate for the study goal (often termed “fit for purpose”) and are transparent about study methods, data collection, curation, and analysis.⁶⁷ Emulating a target trial design⁶⁸ in RWE is helpful for guiding most study designs – be they studies derived from patient registries or follow-up (cohort) studies using clinical or PGHD or RWD available through evidence hubs.

Systematic error (bias) is always a concern, especially as it relates to recruitment and loss-to-follow-up. People willing and able to participate in research studies can be hard to find and may differ from the target population of interest. Studies that rely on volunteer input must keep in mind that, once recruited, participants may soon tire of participation. Losses to follow-up can introduce bias since dropouts may be severely ill or dead. Successful follow-up requires investment and attention to participant engagement, and the impact of misclassification or missing data must be assessed to avoid misinterpretation. A salient example is the potential late mortality risk class signal identified after the approval of paclitaxel-coated balloons and paclitaxel-eluting stents for peripheral artery disease. Interpretation of the increased mortality observed 2 years after device implantation was complicated by missing follow-up information in the data sources used, leading the FDA to convene an advisory panel to evaluate the mortality signal and re-focus efforts on signal detection methods.⁶⁹

Mitigation strategies include blinding outcome assessors, using appropriate statistical analyses to account for measured and unmeasured confounders, quantitative estimation of bias, and small incentives to increase patient participation and retention. That said, it is important to publish study results including detailed descriptions of how patients were recruited and studied, losses to follow-up, etc., so that we may continue to fill in evidence gaps, with a mind to looking both for consistencies and inconsistencies that could be attributed to bias or to differences in baseline study population characteristics. Even imperfect studies can be useful, especially for identifying potential safety signals and in situations where little if any quantitative information is available,²⁷ eg, early studies of the impact of using hand-held mobile phones on the risk of brain tumors.⁷⁰ Such publications may stimulate other descriptive and exploratory research. Consistency in the directionality and relative impact of treatment benefits and risks should increase trust and clinical acceptance.

A number of guidance documents for study design, conduct, and evaluation have been developed that may be of interest to research-minded clinicians, both those that are broadly applicable,^71,72 and those developed with a focus on specific therapeutic areas such as asthma⁷³ and to medical devices.⁷⁴ For those in search of high-level guidance about comparative effectiveness, the GRACE (Good Research for Comparative Effectiveness) checklist for RWE may also be of interest, a stand-out due to its validation rather than having been formulated solely by consensus and its broad applicability.⁷⁵

Conclusion

RCT remains a mainstay for drug development and is increasingly being complemented with RWE to fill in evidence gaps about medical products, services, and interventions as used in real-world settings, including risks and benefits among diverse patients and situations not typically studied in RCT. RWE will play an increasingly important role in clinical, regulatory, and payer decision-making by evaluating the experience of diverse patients and care settings, including quantitative evaluation of long-term benefits, risks, and risk mitigation activities, but the studies need careful design and execution, all tempered by the main study goals. We expect to see more use of linked RWD and supplementation with clinical outcome assessments, wearables, and other PGHD. Success will depend on the contributions of astute clinicians empowered by efficient data capture tools and information provided directly by patients about their experiences, coupled with researchers expert in the design, analysis, and interpretation of RWE.