XVI.B.5. Evaluating the effectiveness of risk minimisation measures

Evaluating the effectiveness of RMM refers to monitoring outcomes (see XVI.A.1.) of routine and additional RMM of a medicinal product. If for a medicinal product no additional RMM are required in the RMP, studies evaluating RMM effectiveness are not mandatory and outcomes of routine RMM are generally monitored through routine pharmacovigilance activities (see GVP Module V) unless agreed otherwise with the competent authority.

XVI.B.5.1. Scope of studies evaluating risk minimisation measures

Any study relating to an authorised medicinal product conducted with the aim of evaluating the effectiveness of RMM is a post-authorisation safety study (PASS) [DIR Art 1 (15)]. For these studies the guidance in GVP Module VIII should be followed in addition to the guidance in this GVP Module and in GVP Module XVI Addendum II on methods for RMM effectiveness evaluation.

Studies evaluating the effectiveness of RMM should be requested by the competent authority for RMM aimed at minimising risks of major patient and public health importance, considering the nature, seriousness and severity of the risk, the magnitude of population exposure and the amount of public concern.

The study objectives should be defined in relation to the intended outcomes of the RMM (see XVI.B.1.2. and XVI.B.5.3.) and consider possible variations in RMM implementation between countries.

Such studies should be designed to provide evidence enabling an evaluation of whether adaptations to the RMM are warranted, including whether additional RMM materials may be discontinued (see XVI.B.6.).

The discussion of the results of an RMM effectiveness evaluation should consider that national variations in RMM implementation and simultaneous events such as changes in clinical guidelines or reimbursement policies and events impacting healthcare (e.g. a pandemic) or media attention may influence the outcomes of RMM.

In certain situations, RMM may lead to unintended consequences, possibly counteracting the effectiveness of RMM. Therefore, evaluating other outcomes beyond the intended ones (see XVI.B.1.2.) may be appropriate upon request by the competent authority. Such unintended outcomes include, but are not limited to, undue burden of RMM on the patient, healthcare professional or healthcare system and unintended changes in using medicinal products (see Table XVI.5.).

XVI.B.5.2. Schedule and documentation of studies evaluating risk minimisation measures

Details of how RMM effectiveness will be evaluated should be included in the RMP (see XVI.C.1.2.).

For a specific RMM, several factors will determine the appropriate timepoints for evaluation, including time since launch or implementation of the RMM (see XVI.B.1.3.), estimated magnitude of exposure, seriousness and severity of the risk and the design of the proposed studies evaluating RMM effectiveness. The following timepoints should be considered by marketing authorisation holders for establishing and agreeing schedules with competent authorities:

• After regulatory implementation an initial evaluation of RMM, e.g. within 12-24 months to allow the possibility of necessary changes in healthcare; and
• Within 4 years of regulatory implementation the overall effectiveness evaluation of the RMM (see XVI.B.5.3.), which, where applicable, can also inform the evaluation of the renewal of a marketing authorisation.

The submissions for studies evaluating RMM effectiveness that fall under PASS category 1 or 3 are to be submitted according to guidance in GVP Modules V and VIII, the EMA Guidance on PostAuthorisation Safety Studies13 and the EMA Post-Authorisation Guidance14 .

Results of any RMM effectiveness evaluation activity should be included in the periodic safety update reports (PSURs) (see XVI.C.1.3.), including a discussion on the need for RMM adaptations (see XVI.B.5.4. and XVI.B.6.).

XVI.B.5.3. Objectives and approaches of studies evaluating risk minimisation measures

In accordance with the scope of RMM effectiveness evaluation (see XVI.B.5.1.), the study objectives include investigating the:

• Extent to which the RMM has been disseminated to the target population(s) as planned (see XVI.B.4.); and,
• Extent to which the RMM has led to the intended knowledge and behaviours in the target population(s), or whether other behavioural outcomes have occurred; and
• Extent, as measurable, to which the intended health outcomes have been achieved within relevant timeframes, or whether other health outcomes have occurred.

Study objectives may differentiate between studying the RMM messages and the individual RMM tool(s) (see XVI.A.1.1.).

Different approaches to data collection and analysis may be applied to investigating each step of the RMM implementation process (see XVI.B.1.3. and Figure XVI.3.), as appropriate.

Measurements and indicators of RMM effectiveness (see XVI.B.5.4.) should be defined in the study protocol; the choice of indicators for success should be duly justified. Marketing authorisation holders and competent authorities may agree on indicators of success to be included in the RMP (see XVI.C.1.2.). Depending on the study objectives, a combination of qualitative and quantitative research methods is recommended. Figure XVI.5. provides an overview of qualitative and quantitative research outcomes that may evaluate the different steps of the implementation process of RMM. Using quantitative measurements (e.g. prescription levels, medicines utilisation patterns, health outcomes) for evaluating the effectiveness of RMM is particularly important and should be considered, where feasible. Qualitative research may be useful for informing the objectives and the conduct of quantitative research, e.g. with regards to the target populations, healthcare or patient home settings and the context of medicines use, and for understanding the reasons for success or failure of RMM (e.g. lack of intended knowledge or behaviour). Such findings may be relevant when regulatory actions for adapting RMM are considered (see XVI.B.5.4. and XVI.B.6.).

The methodological approach should be risk-proportionate and provide results that are meaningful for further regulatory decision-making without placing undue burden on healthcare systems or patients.

XVI.B.5.3.1. Dissemination and knowledge outcomes

Each stage from dissemination of information on RMM to knowledge adoption by the target populations should be considered during RMM effectiveness evaluation (see Figure XVI.3.).

Dissemination methods and individual perceptions of RMM influence the knowledge of the target population about the risk and the intended actions for risk minimisation. Quantitative measurements of the stages of the risk communication process of dissemination and perception may help to understand the dissemination and knowledge adoption and identify ineffective dissemination processes/barriers and knowledge gaps. When used in combination with quantitative research, qualitative measures of the communication process may help to understand factors influencing risk perception and knowledge adoption, including enablers and barriers. Knowledge may be assessed through qualitative research methods involving e.g. semi-guided interviews and/or focus groups, or through quantitative surveys (see GVP Module XVI Addendum II).

Examples of quantitative measurements and qualitative findings to address RMM dissemination and knowledge-related study objectives are provided in Table XVI.6.

XVI.B.5.3.2. Behavioural outcomes

RMM should be evaluated with a view to achieving the intended actions for risk minimisation and other behaviours in medicines use. Factors that may be enablers or barriers for adopted knowledge to result in intended actions are illustrated in Figure XVI.6.. These enablers and barriers may impact on the feasibility of the RMM in practice.

Intended actions for risk minimisation and other behaviours may be evaluated through prescribing, dispensing and other drug utilisation studies, making use of data from electronic healthcare databases or medical records and possibly applying record linkage between different medical and/or demographic data, or through surveys. Quantitative data analyses may also identify enablers or barriers for intended behavioural changes, or study the extent of the impact of enablers or barriers identified through qualitative research.

Examples of quantitative measurements and qualitative findings to address behavioural outcomes of RMM are provided in Table XVI.7.

XVI.B.5.3.3. Health outcomes

Monitoring and investigating measurable health outcomes evaluates whether implemented RMM have achieved the intended patient and public health impact and avoided adverse health outcomes. Changes in health outcomes may only be partially influenced by RMM. Other factors including changes in clinical guidelines or healthcare practices (e.g. therapeutic monitoring) need to be considered. These factors should be identified and assessed where possible as part of RMM effectiveness evaluations.

Examples of quantitative measurements to address health outcomes-related study objectives are provided in Table XVI.8.

XVI.B.5.4. Interpretation of the results of studies evaluating effectiveness of risk minimisation measures

The interpretation of the results of studies evaluating the effectiveness of RMM should support evidence-based decisions as to whether adaptations to RMM are warranted, including whether additional RMM tools may be discontinued. In some instances, important unintended outcomes of RMM (see Table XVI.5.) may warrant regulatory follow-up action (see XVI.B.6.).

National tailoring of RMM agreed at EU level (see XVI.C.3.1.) should be considered in the interpretation of results.

Indicators for success should be determined a priori and specifically for the given RMM. Threshold values may be defined by using e.g. baseline or historical data, expected frequency in similar populations or of similar risks where feasible. Table XVI.9. includes a list of factors for consideration when determining success (or failure) of RMM. The therapeutic/clinical context, local specificities (e.g. clinical guidelines) but also other dimensions (e.g. ethical or societal acceptability) based on e.g. input from patient and healthcare professional organisations (see XVI.B.1.4.) should be taken into account as appropriate.

Where the results of RMM effectiveness studies indicate that e.g. a pre-defined threshold has been met, this suggests that the intended outcomes of the RMM for a specific medicinal product have been achieved. On the other hand, failure to meet a pre-defined threshold may require further investigation to obtain a clear understanding of the reasons that could help explain the failure (e.g. qualitative research, see XVI.Add.II.2.1. and XVI.Add.II.3.1.) and may form the basis to initiate appropriate regulatory action as needed.