Critical appraisal is the process of systematically assessing the value and relevance of scientific research to determine its appropriateness to support product claims and clinical decisions. Clinical trials, systematic reviews and statistical analyses can all be subject to critical appraisal. For the purposes of this article we will focus on critical appraisal of clinical studies.
All clinical evidence is subject to uncertainty. When critically appraising a clinical study, it is key to firstly identify the sources of uncertainty, then assess the level of uncertainty and the impact that uncertainty may have had on study conclusions. Uncertainty in clinical studies can arise from the way the trial was conducted (termed a trial’s internal validity), the way study results are reported, and how applicable study results are to the patients/population for which you wish to draw conclusions (termed a trial’s external validity).
Internal validity
The following should be considered when assessing a trial’s internal validity:
Selection bias occurs when the characteristics of study subjects between treatment groups differ significantly and in a way that could affect the trial outcome. For example, if the treatment group contained more patients with severe disease than the control group, this would likely impact the treatment’s efficacy results. To avoid selection bias, patients should be randomised to different treatment groups using a valid methodology such as centralised computer randomisation. To assess whether selection bias may have impacted study results one should determine how similar study groups are in terms of the variables that may impact on study outcomes.
Detection bias occurs when knowledge of which treatment was received affects the trial outcome. This can be a particular problem in trials where the outcome is patient reported e.g. pain and less of a problem where the outcome is objective e.g. death. Detection bias can be minimised by blinding investigators and study participants. A double blind trial where both investigators and study participants are blinded is the gold standard.
Attrition bias occurs when patients dropping out of the trial leads to study groups becoming imbalanced in terms of patient characteristics, which may impact the trial’s outcome. It can also be a problem if the reason for withdrawing from the trial is linked to unsatisfactory treatment efficacy or intolerable adverse events. An intention to treat analysis (ITT) may mitigate some of the uncertainty over drop-outs.
Reporting bias occurs when study results are selectively reported, for example publishing only interesting or significant results. It may also occur when outcomes are added after the trial i.e. that were not pre-specified. Reporting bias can be a particular problem for systematic reviews, where incorrect conclusions can be drawn based on only positive or negative trial results being published.
Random error is an error that occurs due to chance. All studies are at risk of random error, which can result in a treatment effect being found when in fact there is none or vice versa. This can be more likely to occur if, for example, a trial has few participants the study risks not finding a result where one does in fact exist. To avoid this, all trials should have a sample size calculation and be sufficiently powered (conventionally a probability of <0.05 is set) for the outcome of interest.
External validity
To assess a trial’s external validity one must review the trial in the context of the practice for which one wishes to apply the results.
One must first consider the trial setting and whether there are any differences in the diagnosis or management of the disease between the trial and your practice. For example, one should consider whether the intervention was dosed in a way that could be replicated in practice; whether compliance was higher than would be expected in practice; and whether the comparator and any background medication is relevant to your practice. One must then consider whether any differences between your practice and the trial might have an impact on the effectiveness of the intervention.
The characteristics of patients in the study could also affect trial outcomes. One must therefore consider whether baseline clinical characteristics e.g. disease severity and comorbidities are similar to the characteristics of patients in your practice. For example, if the trial excluded patients at risk of complications it may not be appropriate to assume that the trial results can be applied to patients in your practice who are at risk of complications.
The outcome measures used in the trial should also be considered. If surrogate outcomes were used, are they clinically relevant? If a composite endpoint was used, were all components of the endpoint relevant to your practice?
No trial will be relevant to all patients in all settings. Assessing the external validity of a trial can therefore only be done in the context of the country and type of practice of interest.
Both internal and external validity will form part of an HTA agency’s assessment of the clinical data presented in support of a reimbursement application. It is therefore key that any weaknesses in the clinical data are identified ahead of making a submission so that the impact on the overall validity of the clinical evidence can be assessed.