XVI.Add.II.3. Research methods

Figure XVI.Add.II.1. shows relevant methods and study designs for evaluating the effectiveness of RMM, considering each step of the RMM implementation pathway. Effectiveness evaluation includes measuring intended outcomes of RMM (i.e. product-specific targeted effects) and as appropriate, other relevant outcomes (i.e. non-targeted effects) associated with the use of the concerned and other medicinal products that may counteract the effectiveness of RMM under evaluation (see GVP Module XVI, Figure XVI.3.).

Implementation metrics are useful to determine the extent of RMM implementation as planned and depend on the nature of the RMM. Measures of dissemination and receipt of information and RMM materials are used to ascertain delivery to and receipt by the target audience. Quantitative methods may be applied to assess the implementation and impact of RMM at each implementation step. Qualitative methods assess the context of RMM effectiveness and help determining enablers and barriers in terms of user acceptance and integration of RMM in healthcare systems.

XVI.Add.II.3.1. Qualitative methods

Qualitative research plays a distinctive role in evaluating healthcare interventions (15), especially on issues not yet well understood (9,10). It can study cognitive processes and experiences in their natural setting, such as knowledge, risk awareness, trust, reasoning processes and attitudes about medicines, communication needs and preferences, and experiences of using medicines in real life. Factors that may be enablers and barriers for implementing RMM in healthcare and achieving behavioural change can be identified through qualitative research. These factors include those relating to the interaction between humans and systems elements, as investigated by human factors discipline for enhancing safety and reducing adverse incidences and human error (16–18)9 .

Qualitative studies may generate concepts or hypothesis to be further investigated through quantitative research and inform protocols, sampling strategies and measurement tools for quantitative studies. Qualitative studies may also explore explanations and reasons for results from quantitative research (19) and identify reasons other than the RMM leading to the outcomes of interest. Among the various possible study designs (20), the following are well-established and particularly relevant for evaluating RMM:

• Interpretative phenomenological study: Investigates a phenomenon in the real-world context (21), e.g. the cognitive process or experience of patients and healthcare professionals with disease, medicines use and RMM, including related media behaviours, communication needs and preferences (22).
• Grounded theory study: Aims at developing concepts that are grounded in the data and subsequently formulates – through an iterative and comparative process – a well-grounded theory on a cognitive process or experience, e.g. to explore existing knowledge and beliefs in context of health communication (6,23–25).
• Mixed methods study: Combines qualitative with quantitative methods to benefit from the strengths of each, typically using multiple data sources, perspectives and data analysis methods, for example in an approach called triangulation (5–7).
• Case study: Intends to gain an in-depth understanding of a unique event in its complexity, applying qualitative, quantitative or mixed methods data and analysis, e.g. for understanding experiences of patients and healthcare professionals with RMM for a specific medicinal product, a specific RMM tool or RMM implementation in specific healthcare settings (26,27).
• Action research study: Evaluates ongoing implementation of an action in a participatory approach (6,28), e.g. the implementation of a RMM in healthcare with active research participation of patients and healthcare professionals.

Qualitative studies should be designed for rigour, and tools for assessing their quality are encouraged to be used, in order for the studies to serve as evidence for evaluation and decisionmaking on RMM (10,19,29,30).

XVI.Add.II.3.2. Survey methods

A survey may be conducted to evaluate dissemination of RMM tools, risk knowledge and behavioural outcomes, provided adequate survey methodology is applied.

Sampling and recruitment of survey participants should ensure that the study population is similar to and hence representative of the target population and avoid selection bias due to dissimilarity in one or several relevant aspects. For example, where marketing authorisation holders rely on prescribers to recruit patients, efforts should be made to mitigate the potential for selection bias introduced by e.g. another source for recruiting patients. Selection bias may also occur if webbased survey technology that excludes participants less familiar with internet technology is used.

Bias may be minimised by selecting the optimal sampling frame, accounting for the expected response rate, age, sex, geographical distribution, and additional characteristics of the study population, and by achieving similar response rates across diverse participants to minimise nonresponse bias. As response rates in health surveys are generally low, continuous sampling may be necessary until the pre-defined sample size has been met, and additional measures that improve response rates (31) may be considered.

Bias may also be minimised by assuring that the sample contains appropriate diversity to allow stratification of results by key population characteristics (e.g. by oversampling a small but important subgroup). For example, in a physician survey, the sampling strategy should consider whether a general random sample would be sufficient, or if the sampling frame should be stratified by key characteristics such as specialty, type of practice (e.g. general practitioner, specialist or hospital care). In a patient survey, characteristics such as socio-economic status and education, medical condition(s), and chronic versus acute use of medicines should be considered for optimising the sampling frame.

The recruitment strategy should also consider that accurate and complete data collection is achieved. Efforts should be made to document the proportion of non-responders and their characteristics to evaluate potential effects on the representativeness of the sample.

Surveys often collect and analyse self-reported data, thus introducing misclassification of exposure or recall bias when participants do not remember previous events or experiences accurately or omit details. Respondents may also improve or modify an aspect of their reported behaviour in response to their awareness of being surveyed.

The data collection instrument should be designed to avoid desired-response-bias (e.g. multiplechoice response options with obvious desired response), to cover all relevant aspects of the RMM and to be able to identify different levels of risk knowledge and attitude. For data collection instruments to be considered reliable the following principles should be adhered:

• Pre-testing and validation: Testing the draft instrument in samples of participants that should be similar to the study population identifies questions that are poorly understood, ambiguous, or produce invalid responses. Pre-tests should be carried out using the same procedures that will be used when applying the data collection instrument to the study population.
• Content validity: Items or variables included in the data collection instrument should capture all aspects related to end-users’ risk knowledge and attitudes relevant to the RMM. It is also important that the items or variables are clear and unambiguous and that questions pertaining directly to the implemented regulatory action are avoided (e.g. “do you know that product X is contraindicated for disease Y?”) and non-leading questions are used.
• Construct validity: Items or variables in the data collection instrument should be developed in a way that they are likely to accurately measure (at different degrees) end-users’ risk knowledge and attitudes relevant to the RMM.

Surveys may be analysed quantitatively including:

• Descriptive statistics, such as:
  • Response rate (i.e. proportion of participants who responded of the total number of invited participants);
  • Rate of incomplete responses among responding participants;
  • Pooled proportion of participants responding correctly to the questions;
  • Stratification by selected characteristics such as RMM target population (e.g. healthcare professional or specialist, patient, carer), geographic region, receipt, and type of RMM;
• Comparison of responder and non-responder characteristics (if data is available);
• Comparison of responders and overall RMM target population characteristics;
• Comparison of characteristics of responders with correct and incorrect answers.

Information collected as free text may also be analysed qualitatively, e.g. using thematic content analysis techniques by identifying common recurrent themes or topics.

To obtain valid survey results, a weight may have to be attached to each respondent considering the following:

• Differences in selection, e.g. if certain subgroups were over-sampled;
• Differences in response rates between sub-groups;
• Differences of responders compared to target population (e.g. healthcare speciality, volume of prescribing);
• Clustering

Variations among healthcare settings in Member States may pose challenges to implementing survey studies in several Member States due to time constrains for determining and complying with national ethical and data protection requirements. Therefore, early feasibility assessment is paramount in the successful implementation of a survey. National (or regional) requirements for providing incentives to survey participants also need to be accounted for.

There may be also data protection requirements when healthcare professionals are contacted based on a prescriber list of a marketing authorisation holder.

Although survey studies aimed at evaluating risk knowledge and attitudes do not attempt to collect patient health-related information, patients who complete the survey are likely to have received the medicinal product revealing their condition/disease. Therefore, unless the patient response is completely anonymous, data protection regulation applies, and informed consent must be provided.

Survey studies must follow the provisions of the legislation on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, as laid down in Regulation (EU) 2016/679 (General Data Protection Regulation) and Regulation (EU) 2018/1725 of the European Parliament and of the Council and require approval(s) by the relevant body(ies) in Member States.

XVI.Add.II.3.3. Pharmacoepidemiological methods

Outcomes of RMM may investigated with non-interventional methods that measure how medicinal products are prescribed, dispensed, or used over time, by means of electronic health records, medical chart abstraction or claims data (see XVI.Add.II.2). Detecting changes in adverse reaction reporting, despite known limitations, may contribute to this investigation (see XVI.Add.II.2.7. ).

Since RMM are generally implemented in the entire target population, the identification of a control group may not always be possible and the comparison against suitable reference values should be considered (see GVP Module XVI).

For marketed medicinal products, quantitative measures (see GVP Module XVI) should be estimated in the same study population before and after the RMM intervention, with preintervention information acting as a surrogate control (i.e. quasi-experimental designs). However, in absence of pre-intervention information (e.g. for medicinal products with RMM at the time of initial marketing authorisation), any effect of the RMM estimated at (a) time point(s) after implementation can only be evaluated against a predefined reference value (i.e. literature review, historical data, expected frequency in general population, outcome frequency in pre-authorisation clinical trials) taking into account all possible limitations (32) (see GVP Module XVI). The selection of a reference value should be justified.

Whilst appropriate to describe the population for understanding generalisability of observed outcomes, simple descriptive approaches do not determine whether statistically significant changes have occurred (3,33).

XVI.Add.II.3.3.1. Single time point cross-sectional study

The guidance on cross-sectional study designs in GVP Module VIII applies. Cross-sectional studies can only measure an association between exposure and outcome at a single point in time. Therefore, the method is commonly used to monitor indicators of RMM implementation and to complement other studies investigating e.g. patterns of medicines use.

XVI.Add.II.3.3.2. Before-after cross-sectional study

A before-after cross-sectional study is defined as an evaluation at one point in time before and one point in time after the RMM dissemination to healthcare systems (accounting for the implementation timeframe). Including a control can strengthen this design (3); however, careful consideration should be given to whether a suitable control can be identified, e.g. healthcare professionals not targeted by the RMM to control for general prescribing trends. When uncontrolled, baseline trends are ignored, potentially leading to RMM outcomes being estimated incorrectly.

When a RMM is put in place at the time of initial marketing authorisation, the comparison of an outcome frequency indicator obtained after the RMM against a predefined reference value would be acceptable (see GVP Module XVI).

XVI.Add.II.3.3.3. Before-after time series analysis

Time series analysis has commonly been used to evaluate the effectiveness of RMM and should be considered whenever feasible as one of the more robust approaches (3). A time series analysis spanning the date of RMM dissemination to healthcare systems (e.g. interrupted segmented regression analysis) accounts for secular trends and can provide statistical evidence about whether observed changes are significant.

Time series analysis is well suited to study changes in outcomes that are expected to occur relatively quickly following RMM, such as changing prescribing rates. Time series analysis can be used to estimate the immediate change in outcome after the RMM, the change in trend in the outcome over time compared to before, and the effects at specific time points following the RMM. The Cochrane Effective Practice and Organisation of Care (EPOC) Resources for Review Authors on Interrupted Time Series (ITS) Analysis10 provides further information on the utility of time series regression (34).

Time series analysis requires that enough data points are collected before and after the RMM. The power to undertake a time series analysis depends upon the sample size, the effect size, the prevalence of exposure, the number of data points and their balance before and after the intervention time period (35). Long time periods may also be affected by changes in trends unrelated to the RMM that can violate model assumptions and introduce confounding when evaluating RMM.

Like the before-after cross-sectional design, including a control can strengthen this design by minimising potential confounding.

Factors such as autocorrelation, seasonality and non-stationarity should be checked when conducting time series analysis and may require more complicated modelling approaches if detected or considered likely to occur (36). Interventions associated with major immediate changes (e.g. product withdrawals) may be evaluated without regression modelling, but they risk producing spurious results when the changes are more subtle or multiple confounders are present (3).

Time series analysis also requires that the time point of RMM dissemination to healthcare systems (accounting for the implementation timeframe) is known prior to the analysis. When this is not the case (e.g. during a phased roll out of a regulatory action), more complex modelling techniques and data-driven time series approaches such as Joinpoint regression analysis could be considered (37). There are literature examples of time series analysis using a control (38), estimating effects twelve months after the intervention (33), dealing with autocorrelation and seasonality (39), and using Joinpoint regression (40).

XVI.Add.II.3.3.4. Cohort study

The cohort study design may be useful to establish the base population for the conduct of drug utilisation studies to assess behavioural and health outcomes (see GVP Module XVI) or to perform aetiological studies (see GVP Module VIII).

Cohort studies are in particular suitable to examine RMM aimed at preventing adverse pregnancy outcomes (41) or other effects on health outcomes, or medicines use in populations targeted by the RMM (42). Modelling the effect of RMM on health outcomes may require more complex study designs. In aetiological studies, propensity score methodology may be used, e.g. to measure the reduction in stroke with warnings on the use of antipsychotics (43).