RCT or die?

How do we prove beyond the shadow of a doubt that HIPPY

has made a significant difference in the futures of these children?

OK, that headline sounded a bit dire and dramatic I know but my mind keeps wandering back to the heated discussions we had on RCTs during the National HIPPY Evaluation Summit. Randomized Control Trials (RCTs) had everyone avidly sharing their opinion both for and against. Let's see if I can outline some thoughts while still maintaining some neutrality. Almost all of social science is built on quasi-experimental designs as well as in depth qualitative studies (such as case studies). RCTs follow the medical model for studying the effectiveness of drugs or medical interventions. In an RCT, participants are randomly assigned to a control group or an intervention group and evaluation is done on both groups throughout and/or after the intervention. Recently this approach is being discussed among legislators and policy-makers and held up as the “gold standard” of social science research. Why aren’t we hopping quickly onto this bandwagon? Why haven’t we done this already? First of all, it’s a very expensive evaluation design that requires serious resources in terms of expertise and staffing that makes it inaccessible to most programs—it necessitates a separate source of funding for the evaluation component alone that is most likely more expensive than the cost of the program it evaluates. Second, it works best in the initial implementation phase of a program, not as an ongoing program evaluation design. This reduces the opportunity to implement RCT to new programs, which may be at their most vulnerable in terms of fidelity to the model due to the inexperience of staff. The third and last reason why we’ve been hesitant to implement RCT is the nature of the HIPPY intervention. As a voluntary program being offered to vulnerable populations we are recruiting families based on relationships we’re building with them. During this recruitment phase we discuss with parents how their child will benefit from the parent’s commitment to the weekly visit and to working with their child daily. We discuss the materials they will be provided and the support they will be offered through home visits and group meetings. In an RTC model we would then have to randomly assign parents to the HIPPY and control groups. While this is common practice in drug trials, it’s not common in social services and could be damaging to the credibility and standing of the sponsoring organization if not handled in a socially responsible and positive way. If we could work out those three “little details” we would consider hosting a HIPPY RCT.