The hidden dangers of randomized controlled trials The need to demonstrate the effectiveness of nonprofit social programs has led to a rapid rise in the use of randomized controlled trials (RCTs), for evaluation. As a result, most nonprofit sector professionals can tell you why nonprofits should do an RCT. This book tells you why they probably shouldn't, and what to do instead. Mismeasuring Impact explores why RCTs are being embraced as the "gold standard" for nonprofit evaluation, despite the high cost and time investment required and the serious problems with using RCTs in a nonprofit context. (Most RCTs conducted in nonprofits fail to meet required standards for rigor, undercutting their accuracy). The book describes what happens inside nonprofits when they take part in RCTs, the unintended equity issues that arise, and why nonprofits feel pressured participate in RCTs despite the problems. University of Chicago professors Marwell and Mosley's research is based on extensive interviews with key players: nonprofit managers, professional program evaluators, and program officers in philanthropic foundations. The book argues that, ultimately, RCTs are used to poorly ground nonprofit legitimacy, not to foster nonprofit improvement. RCTs also privilege program and organizational standardization over the key strengths of nonprofit organizations: flexible innovation and responsiveness to community needs. Nonprofits and funders need forms of evaluation that lift up these strengths. Mismeasuring Impact offers alternative approaches that build strong organizations, not just cookie-cutter programs, and which funders and nonprofits of all sizes can support.
Nicole P. Marwell and Jennifer E. Mosley are Professors at the Crown Family School of Social Work, Policy, and Practice at the University of Chicago. Their research on nonprofit organizations has been published widely in leading journals in the fields of nonprofit studies, sociology, public administration, and social work.
Acknowledgments 1. FIVE PROBLEMS WITH RCTs, AND WHY THEY THREATEN THE NONPROFIT SECTOR The Case for RCTs Are RCTs Really the "Gold Standard" for Nonprofits? Five Problems with Using RCTs in Nonprofit Organizations Goals of This Book Brief Answers to Some Common Objections to Our Argument 2. THE SIREN SONG OF RCTs: BUILDING THE GOLD STANDARD MOVEMENT How Did We Get Here? Understanding the Growth of RCTs for Social Programs The Evidence Battle The Funding Battle Spreading RCTs to U.S. Nonprofits: The Social Innovation Fund Changing the Conversation 3. NULL EFFECTS AND DUMB EFFECTS: DO RCTs REALLY TELL US "WHAT WORKS?" The Ubiquity of Null Effects Dumb Effects: When Positive Effects Are Not Very Meaningful Moving Beyond RCTs 4. "THIS WAS HARDER THAN WE THOUGHT": IMPLEMENTING RCTs IN NONPROFITS Twice the People, Twice the Work: RCT Recruiting Challenges Staff Capacity and Organizational Mission "It Sounded a Lot Easier Than It Actually Was" "It Sounded a Lot Easier Than It Actually Was" Delays and (Financial) Drains RCTs Don't Just Evaluate, They Can Change Both Programs and Organizations Putting Organizational Well-Being First 5. LOOKING FOR LEGITIMACY: WHY WE KEEP DOING RCTs RCTs Can Build Organizational Legitimacy Does an RCT Always Enhance Nonprofit Legitimacy? Evaluators See Limits on RCTs The View from Philanthropy RCTs May Increase Legitimacy, but Not Without Consequences 6. WHAT KIND OF WORK SHOULD WE VALUE? HOW RCTs UNDERMINE THE EQUITY GOALS OF NONPROFITS AND THE COMMUNITIES THEY SERVE Is Pursuing Effectiveness the Same as Pursuing Equity? Do RCTs Privilege Evidence-Building Over People? The Rich Get Richer Practicing Equity: A Key Goal for Philanthropy Evaluation, Effectiveness, and Equity 7. A BAD FIT :HOW RCTs HINDER NONPROFIT RESPONSIVENESS AND INNOVATION Long Time Horizon The "Hold Still" Challenge: RCTs Limit Ongoing Program Responsiveness and Innovation "You Get One Shot" A Black Box The Need for Alternatives to the RCT . . . and More Evaluation Capacity Overall Evaluation Should Better Support Responsiveness and Innovation 8. HOW MIGHT WE THINK ABOUT EVALUATION DIFFERENTLY? THREE PRINCIPLES FOR NONPROFIT IMPROVEMENT Principle 1: Tailor Evaluation to Specific Organizational Strategies and Community Needs Principle 2: Centering Participant Perspectives Is Key to Nonprofit Performance Principle 3: Focus on Improvement Through Iteratively Addressing Problems Instead of Standardizing Solutions An Improvement Orientation 9. MOVING BEYOND RCTs: LESSONS FOR NONPROFIT LEADERS, DONORS, EVALUATORS, AND POLICYMAKERS Evidence Matters Five Problems with RCTs When Is the Right Time to Do an RCT? Three Principles for Using Evaluation to Foster Nonprofit Improvement Recommendations for Building Evidence Outside the Evidence Hierarchy The Way Forward Appendix: Research Methodology Notes Bibliography Index
"Marwell and Mosley challenge the prevailing myth that RCTs are the 'gold standard' for evaluation, revealing RCTs as a movement rather than an infallible method. A must-read for nonprofit managers, funders, and anyone seeking a deeper, more nuanced approach to measuring impact." -Johanna Mair, Professor of Organization, Strategy and Leadership at the Hertie School "Finally, a book that puts RCTs in their proper place: an evaluation method that's useful only under limited conditions. Marwell and Mosley show how the uncritical use of RCTs can harm nonprofits-by undermining relationships with communities, sapping staff morale, and putting organizations at risk. The gold standard is now, more fittingly, just a hammer in the evaluation toolbox." -Alnoor Ebrahim, Professor of Management at Tufts University "In an exemplary work of humane and rigorous scholarship, Marwell and Mosley stand athwart the rise of the RCT as the dominant form of evidence for nonprofit impact and yell: 'Stop!' Even if you are a devotee of the RCT-especially if you one-this is an essential book to learn from and wrestle with." -Benjamin Soskis, Center on Nonprofits and Philanthropy, The Urban Institute "A courageous, timely, and well-written warning against fashionable methodological assumptions in nonprofit evaluation. Marwell and Mosley demonstrate the not-so-glittering consequences of the so-called gold standard in evaluation." -Peter Dahler-Larsen, University of Copenhagen. Author of The Evaluation Society. "Though I did not agree with everything here, this book should be required reading for thoughtful evaluators who want to 'do no harm' and desire to adopt the right tool in judicious partnership with funders, program operators, and community members." -Matthew Stagner, Former president of the Association for Public Policy Analysis and Management "Rather than succumbing to pressures to embrace the Randomized Controlled Trial (RCT) 'gold standard,' this book sets the record straight with a clear description of how we got here and guidance for more effective evaluation. Essential reading for social sector leaders who want to 'do good better,' and social scientists who want to help them." -Jodi Sandfort, Dean at the Evans School of Public Policy and Governance, at the University of Washington "Accurately measuring the impact of nonprofit organizations remains the 'Holy Grail' for many. Yet, there are many reasons why, all these years on, we are still short of consensus on the best way to do that work. Marwell & Mosley add to our shared understanding of how one such method - Randomized Controlled Trials - may not be the 'gold standard' once advertised. While measuring impact is important, we must not lose sight of the inputs needed to ensure nonprofits are healthy and equipped to deliver the impact we all desire." -Jeffrey Moore, Chief Strategy Officer at Independent Sector "Marwell and Mosley show that RCTs cannot capture the complexity of the real world and undermine important nonprofit contributions to communities and society as a result. Mismeasuring Impact is necessary reading for scholars doing policy-relevant research and for policymakers who want to base their decisions on evidence." -Mary Pattillo, author of Black on the Block: The Politics of Race and Class in the City, Harold Washington Professor of Sociology at Northwestern University "RCT's have quickly emerged as the preferred evaluation tool for many nonprofit programs. Yet Marwell and Mosley, in their timely, outstanding book, persuasively call attention to the programmatic and organizational drawbacks of RCTs that can undermine nonprofit effectiveness, responsiveness, and capacity for innovation." -Steven Rathgeb Smith, McCourt School of Public Policy, Georgetown University, past president of the Association for Research on Nonprofit and Voluntary Action (ARNOVA) and the International Society for Third Sector Research (ISTR) "As experts in sociology, social work, and the nonprofit sector, the authors offer an illuminating and sobering critique of randomized controlled trials (RCTs) for charitable organizations. This book will appeal to academics, policymakers, and practitioners alike." -David Suarez, Colleen Willoughby Endowed Faculty Fellow in Philanthropy & Civil Society, Evans School of Public Policy and Governance, at the University of Washington
