Looking beyond randomization to estimate the effects of interventions in RCTs

This month’s contribution comes from Sinziana Oncioiu, who is a public health researcher affiliated with Karolinska Institutet in Stockholm, Sweden. In this article, she will provide an in-depth delineation of her research which she presented at the EUSPR conference in Ljubljana in 2015. Her research targets complementary analyses to intention-to-treat analysis in an effectiveness study that was conducted at Karolinska Institutet.    

At the 6th annual conference of the European Society for Prevention Research (EUSPR), I had the chance to present the results of my Master’s thesis. Its focus was on alternative methods to intention-to-treat (ITT) for a better understanding of the effects of interventions evaluated using randomized controlled trials (RCTs). The data I worked with came from the FRITT study which stands for “Free from tobacco in dentistry”. It is an RCT designed to evaluate the effectiveness of a brief counselling on tobacco cessation in dental clinics in Sweden1. The ITT analysis showed that the brief advice was effective in reducing the amount of tobacco used at baseline, but not in achieving abstinence1. In this post, I will walk you through the main lessons learnt during this study.

What do we actually estimate using ITT?

Briefly, the answer would be: the effect of the assigned intervention. ITT is the usual, gold standard approach taken to analyse RCT data. Through this method, we compare the outcomes between patients who were randomly assigned to receive the intervention with those randomly assigned to the reference condition (usual care, for instance). In our case, assignment to receive the brief advice from the dental practitioner increased, on average, twice the odds of halving the baseline tobacco consumption compared with assignment to usual care. If all the dental practitioners were to comply in delivering the care corresponding to their assignment, then the effect of the assigned intervention would be exactly the same as the effect of the intervention actually delivered. However, in real life situations, 100% adherence to the protocol is unlikely to be achieved; therefore the ITT estimation will no longer be equivalent with the effect of the intervention delivered. The secondary analyses I performed (per-protocol, as-treated and instrumental variable analyses) tried to unveil the effect of the delivered intervention by taking non-adherence into account.

Secondary analyses: comparisons of hidden subgroups in the RCT population

Once you start to analyse the RCT population in the light of non-adherence, new subgroups emerge apart from the intervention and control group: the dental practitioners who adhere to the protocol and those who do not. Going even further, we could classify the dental practitioners in theoretical groups that we cannot actually observe; for instance those who adhered by chance. They would have never delivered the brief advice according to the protocol, but because they were assigned to the control group, we classify them as compliers. If they had instead been assigned to the intervention group, we would have classified them as non-compliers. The secondary analyses were based on comparisons across these different subgroups.

  • In the per-protocol analysis, we compared the outcomes of the patients treated by the dental practitioners who gave the intervention as described in the protocol with those of the patients who received usual care (in our case, the entire control group).
  • The as-treated analysis disregarded randomization, with the population being perceived as a cohort in which patients were classified as “exposed” if their dental practitioner followed the protocol and “unexposed” if their dental practitioner did not follow the protocol or if they received usual care.
  • The instrumental variable analysis was used to compare subgroups of patients whose dental practitioners would always act in the same way: comply with their assignment, irrespective of which the assignment was. We try to compare groups that would be as similar as possible by discounting for the effect among those who happen to adhere just by chance: i.e. patients treated by dental practitioner who would never give the treatment, but appear to be compliers just because they were in the control group.

The aforementioned methods are useful for estimating the effect of the treatment actually delivered by the dental practitioner. However, selection bias and confounding are big threats to the validity of the results obtained from per-protocol and as-treated analyses respectively2,3,4. If several assumptions are met, instrumental variable analysis is deemed to estimate the effect of the intervention actually received in a less biased way by implicitly controlling for confounding5. In our study, the per-protocol analysis closely replicated the results of intention-to-treat analysis, while as-treated and instrumental variable analyses posed application problems and yielded partially biased results.

The main challenges

When applying these secondary analysis methods, I encountered several challenges. Nonetheless, the main challenges came from the difficulty of defining adherence or “clinically” relevant adherence when the intervention addresses behavioural change. It is noteworthy that the majority of the literature on these secondary analyses methods is based on studies of pharmacological treatment, where defining the therapeutic dose is a more straightforward and objective endeavour. For the current analyses, I used the most stringent definition of adherence identified in the literature. However, this left little room for any adaptation of the protocol, but adaptation of the protocol is expected to happen anyway, when the treatment is a “structured conversation” between the dental practitioner and the patient. Therefore, patients who received very sound structured counseling, but for which not all the steps described in the protocol could be ticked, were classified as “untreated”. This strict classification may explain the fact that the “untreated” patients in the intervention group had more favorable outcomes than those receiving “usual care”. Most probably, even if not delivering the intervention “as-intended”, the dental practitioners of the “untreated” patients in the intervention group offered counseling which differed in intensity and quality from “usual care”.

It can be fairly cumbersome to define usual care. Usual care is not a real assignment; dental practitioners have no other option than to be compliant with your own “usual care”. In some cases, the usual care delivered by the dental practitioners presented all the elements of the intervention we were evaluating. What should we do with those cases? They belong to the “usual care” group, but the “usual care” provided matches totally the intervention under study. Given that the frequency of these cases was low and the initial interest was to evaluate the intervention against usual care, we assumed that all practitioners in the control group were fully compliant with the usual care. Further research and methodological advancement are needed to provide a better understanding on proper ways to deal with “usual care” as a control condition when performing per-protocol, as-treated and instrumental variable analyses.

Concluding remarks and importance of questioning ITT

Strict monitoring of the delivery of the intervention both in intervention and control group is essential for estimating adherence to the protocol. Moreover, measuring carefully all the potential confounders at baseline is extremely important for reasoning about the validity of the results of the secondary analyses discussed above. I was lucky to perform these secondary analyses on data coming from a study in which both the monitoring and the measurement of the confounders had been rigorously done.

Generally, if adherence to the protocol and participation rate are high, then most likely ITT would offer a good estimation of the true effectiveness of the intervention, like it happened in our study. However, defining non-adherence and performing secondary analyses was important to strengthen and put in context the inference based on ITT analysis.

Acknowledgements

The research project I presented above was done under the supervision of Professor Maria Rosaria Galanti with the contribution of Suvi Virtanen, Livia Franchetti-Pardo and Fabrizio Faggiano.

About the author:

My name is Sinziana Oncioiu and I am a public health researcher affiliated with Karolinska Institutet in Stockholm, Sweden. I am a pharmacist by training with a Master’s degree in Public Health Epidemiology. I am willing to deepen my knowledge in epidemiology focusing on prevention of mental health problems through a life course approach. I have a special interest in the impact of social change on health in Central and Eastern Europe as I come originally from Romania. Feel free to contact me for any further information at sinziana-ioana.oncioiu[at]ki.se.

Bibliographic references:

  1. Virtanen, S. E., Zeebari, Z., Rohyo, I., & Galanti, M. R. (2015). Evaluation of a brief counseling for tobacco cessation in dental clinics among Swedish smokers and snus users. A cluster randomized controlled trial (the FRITT study). Preventive Medicine, 70, 26–32. http://doi.org/10.1016/j.ypmed.2014.11.005 []
  2. Shrier, I., Steele, R. J., Verhagen, E., Herbert, R., Riddell, C. A., & Kaufman, J. S. (2014). Beyond intention to treat: what is the right question? Clinical Trials (London, England), 11(1), 28–37. http://doi.org/10.1177/1740774513504151 []
  3. Stuart, E. A., Perry, D. F., Le, H.-N., & Ialongo, N. S. (2008). Estimating intervention effects of prevention programs: accounting for noncompliance. Prevention Science: The Official Journal of the Society for Prevention Research, 9(4), 288–298. http://doi.org/10.1007/s11121-008-0104-y []
  4. Hernán, M. A., & Hernández-Díaz, S. (2012). Beyond the intention-to-treat in comparative effectiveness research. Clinical Trials (London, England), 9(1), 48–55. http://doi.org/10.1177/1740774511420743 []
  5. Glymour, M. M. (2009, November). Natural Experiments and Instrumental Variables Analyses in Social Epidemiology. Retrieved from http://www.tc.umn.edu/~alonso/Glymour_2006.pdf []

Leave a Reply

Your email address will not be published. Required fields are marked *