Article  |  Reentry

BetaGov Supports Practitioners and Evaluators in Conducting Randomized Control Trials to Test Criminal Justice Programs

 |   | 

This article was originally published in Translational Criminology, the Magazine of the Center for Evidence-Based Crime Policy, George Mason University, available at: Reprinted with permission.

Many fields employ researchers and others who routinely conduct experiments using randomized control trials (RCTs) to test daily operations to learn about and improve their work to increase success. However, fields in the public sector, including criminal justice, have lagged in experimentation. Testing is pervasive in medicine, marketing, and business as part of the standard process for decision-making and advancement. Amazon CEO Jeff Bezos said his company’s success is “a function of how many experiments [they] do per year, per month, per week, per day.” [1] Medicine, marketing, and business leaders recognize how rare it is to get it right the first time, and they embrace failure as a learning opportunity.[2] In the public sector, ethical concerns, practicality, and a lack of knowledge are some of the barriers to the use of RCTs.[3]

Most public policies and practices—such as how we educate our children, deliver health care, rehabilitate convicted offenders, or house the homeless—have one thing in common: They have never been empirically tested. Conducting rigorous evaluation traditionally involves academic researchers and government funding and requires navigating through bureaucratic red tape that makes the research difficult to accomplish. As a result, many commonplace policies intended to make citizens smarter, safer, or healthier are based more on public perception or “business as usual” than on empirical data.

Randomized Control Trials: Why Are They Important?

Randomized control trials are the scientific gold standard for program evaluation. Evaluators employ RCTs to measure program effectiveness by isolating the effects of programmatic conditions from other factors that may contribute to varying outcomes among similar groups, such as systematic bias in program participation.[4] RCTs can help ensure government policies are effective and will help those who need them most.[5] Programs that undergo RCT-based evaluations also become more attractive to grantmakers and investors who value scientific evidence. Robust and stable funding streams enable program adoption, expansion, and long-term improvements.

What Is BetaGov?

BetaGov is a multidisciplinary group of academic and practice-experienced researchers who help agencies explore possible improvements in policies and practices in domains ranging from criminal justice to education. BetaGov’s approach is unconventional by design; BetaGov exists because conventional approaches to conceptualizing and conducting research to test policies and practices can be inefficient, and the results are often irrelevant to real-world practitioners and policymakers. BetaGov was created to promote scientific evaluations of policies and practices via RCTs and to make these assessments the norm rather than the exception. BetaGov’s mission is to help agencies, policymakers, and others develop, conduct, analyze, and share research on policies and practices that affect the public they serve. Guidance from BetaGov—provided at no cost to the end user—facilitates design and implementation of research conducted by service agencies and departments at all government levels. The goal is to significantly increase the pace of learning about policies pertaining to health services, social services, criminal justice, education, and other domains; identify promising innovations; and identify policies and practices that are inefficient or ineffective.

With BetaGov’s guidance and assistance, practitioners can carry out their own RCTs. Being able to design and implement a trial without funding and often without regulatory hurdles means that the trial can be more quickly conducted and completed. The private sector has long relied on simple, pragmatic RCTs to improve efficiency and performance; BetaGov promotes the use of these same techniques to inform policy solutions for the most challenging health and social problems.

A BetaGov Collaboration Example: Randomized Control Trial of an Illinois Reentry Program

The Illinois Criminal Justice Information Authority (ICJIA) is a state government agency that administers federal criminal justice grants and serves as the state’s Statistical Analysis Center for criminal justice research. Researchers from ICJIA were recruited to evaluate a newly established 2018 prisoner reentry program in Illinois. The program, Pathway to Enterprise for Returning Citizens (PERC), offers entrepreneurship training to individuals returning to Cook County, Illinois, communities from prison. The training focuses on how to start a business, offers a mentor for support, and provides an opportunity to obtain a small business loan. There is very little research on prior entrepreneurship reentry programs and none employing an RCT design.

The ICJIA researchers were tasked with measuring program benefits.They sought to collect program process information[6] and use an RCT to compare program outcomes, such as securing employment, reducing recidivism (arrest and reincarceration), and starting a business. As a government agency with research experience but few prior opportunities to apply an RCT, ICJIA partnered with BetaGov for assistance. BetaGov and ICJIA researchers scheduled regular conference calls to discuss evaluation components and RCT implementation. The team stratified applicants by prison release date and distance from PERC training agencies, and BetaGov completed the random assignment of program applicants into either the treatment group (PERC-trained) or the control condition (on parole with no PERC training). With BetaGov’s help, ICJIA researchers saved time while avoiding the appearance of possible bias in participant selection.

Lessons Learned from Our Randomized Control Trial

Practitioners Should View Evaluation as a Valuable Tool

To seasoned practitioners with years of field experience, a formal evaluation guided by researchers may seem irrelevant. After all, most programs make adjustments over the course of development, and changes on the ground are easily recognized. However, even the sharpest practitioners may not recognize that complex external factors unrelated to programmatic decisions may be driving observable changes. The scientific method employed by RCTs can most effectively rule out external factors as the explanation for ground-level changes, which makes an RCT evaluation an efficient and accurate way to discover whether program activities achieve the desired outcomes. Accordingly, it saves time and resources otherwise spent on ineffective and unproven modifications.

Of course, researchers must acknowledge that practitioners often have more immediate needs. Short-term feedback, interim reports, program updates, and troubleshooting may be required long before the formal evaluation is completed. Researchers should confer with practitioners on their needs to see what can be immediately addressed and clearly communicate the project scope to manage expectations. The development of methodology, data collection, and other activities required by rigorous evaluation may take longer than what is naturally learned throughout normal operations. However, knowledge gained through an RCT evaluation will be based on empirically derived data and useful in the long-term.

Examine a Program’s Mission and Goals

For an evaluation to be a truly collaborative endeavor, the mission of a program must be reflected in the evaluation design and in the outcomes to be measured. Goals of funders, program leadership, staff, and other core stakeholders must be aligned and sufficiently understood to ensure meaningful research questions and valuable results. Establishing clear goals can be difficult for new programs and for programs lacking consistent direction from leaders, but an evaluation can be a great opportunity to define new goals or get reacquainted with original program goals. Researchers can help construct operational definitions to accurately measure the concepts most meaningful to program stakeholders.

Creating a logic model that maps how stakeholders are connected to program activities and how those activities are connected to main goals is a great way to align program partners. An honest, open-minded discussion about the feasibility of the program model should take place. If the purpose and scope of a program or its evaluation is not clear to key stakeholders before the evaluation begins, more organization and collaboration is needed. In the end, practitioners and researchers should be able to provide similar answers to the questions: “What does the program hope to achieve?” and “How will you know if that has been achieved?”

Know How to Address Ethical Concerns and Design Issues

An RCT can be derailed by data collection barriers, mid-program adjustments, scheduling, and study participant retention issues. A plan should be in place to deal with these issues as they arise. However, sometimes the very nature of an RCT can be problematic. Some stakeholders may have concerns about randomly excluding people from a program meant to be helpful. However, assigning people to not receive a beneficial treatment or intervention can be acceptable under the right circumstances. No one sets out to create an ineffective program, let alone one that produces unintended negative consequences for its participants. Good intentions, however, do not make good programs. Program staff need to be open to the idea that their programs as currently administered may be ineffective, inefficient, or unhelpful, especially when their program is new or untested. The purpose of a rigorous evaluation is to determine whether a program does what it aims to do so that participants benefit as intended.

Researchers must keep the principles of ethical research in focus at all times and understand that scientific research has not always embodied these principles in the past.[7] Collaboration on RCTs will work best when researchers directly address practitioner concerns and can offer practical solutions to issues with random assignment when possible, including assigning from a waitlist or providing alternate programming over no programming. [8] Finally, researchers should recognize when the logistics and context of a project are simply not conducive to an RCT, such as when quick results are paramount, precision or causal inference are not desired, proper planning is impossible, or if an RCT would introduce too many complications for a budding program to handle.[9]

Be Resilient and Continue to Advocate for RCTs in Criminal Justice

Not every part of an RCT evaluation will go smoothly, but when researchers are committed to a solid evaluation plan, adjustments can be made to account for arising issues. This can be easier with the assistance of BetaGov and other associations whose expertise and experiences allow them to help solve evaluation problems associated with unexpected events, strained resources, and knowledge gaps. With supportive services that speak the language of both practice and research, researchers should feel encouraged to advocate for more RCTs in the public sector and examine the questions that programs really need answered. Only rigorous evaluations that ask meaningful questions will result in evidence-based programming and sustainable improvement.

This project was supported by Grant #16-DJ-BX-0083, awarded to the Illinois Criminal Justice Information Authority by the U.S. Department of Justice Office of Justice Programs’ Bureau of Justice Assistance. Points of view or opinions contained within this document are those of the authors and do not necessarily represent the official position or policies of the U.S. Department of Justice.

Suggested citation: Escamilla, J., Reichert, J., Hillhouse, M., & Hawken, A. (2018). BetaGov supports practitioners and evaluators in conducting randomized control trials to test criminal justice programs. Translational Criminology, 15, 29-31.

  1. Simmons, M. (2017). Forget the 10,000-hour rule; Edison, Bezos, and Zuckerberg follow the 10,000 experiment rule. Medium. ↩︎

  2. Edmondson, A. C. (2011). Strategies for learning from failure. Harvard Business Review. Retrieved from ↩︎

  3. Shadish, W. R., Cook, T. D., & Campbell, D. T. (2001). Experimental and quasi-experimental designs for generalized causal inference. Boston, MA: Houghton Mifflin. ↩︎

  4. Shadish, W. R., Cook, T. D., & Campbell, D. T. (2001). Experimental and quasi-experimental designs for generalized causal inference. Boston, MA: Houghton Mifflin. ↩︎

  5. Buck, S., & McGee, J. (2015, July). Why government needs more randomized controlled trials: Refuting the myths. Laura and John Arnold Foundation. Retrieved from ↩︎

  6. Data included intake data, pre-tests and post-tests, exit surveys, mentor surveys, focus groups, and interviews. ↩︎

  7. Mandal, J., Acharya, S., & Parija, S. C. (2011). Ethics in human research. Tropical Parasitology, 1(1), 2–3. The Belmont Report. (1979, April). Washington, DC: Department of Health, Education, and Welfare. Retrieved from ↩︎

  8. Shadish, W. R., Cook, T. D., & Campbell, D. T. (2001). Experimental and quasi-experimental designs for generalized causal inference. Boston, MA: Houghton Mifflin. ↩︎

  9. Shadish, W. R., Cook, T. D., & Campbell, D. T. (2001). Experimental and quasi-experimental designs for generalized causal inference. Boston, MA: Houghton Mifflin. ↩︎