Kylie Hutchinson 
Evaluation Failures 
22 Tales of Mistakes Made and Lessons Learned

‘This is the evaluation book we′ve been waiting for! A must-read for all learning and working in the field.’ 

–Amanda M. Olejarski, West Chester University




Evaluation Failures: 22 Tales of Mistakes Made and Lessons Learned is a candid collection of stories from seasoned evaluators from a variety of sectors sharing professional mistakes they have made in the past, and what they learned moving forward. As the only book of its kind, editor Kylie Hutchinson has collected a series of engaging, real-life examples that are both entertaining and informative. Each story offers universal lessons as takeaways, and discussion questions for reflective practice. The book is the perfect companion to anyone working in the evaluation field, and to instructors of program evaluation courses who want to bring the real world into their classroom.

Table of Content

Foreword – Michael Quinn Patton

Acknowledgments

Introduction

PART I. MANAGE THE EVALUATION

Chapter 1. It’s Not Me, It’s You: The Value of Addressing Conflict Head On – Corey Newhouse

Chapter 2. The Scope Creep Train Wreck: How Responsive Evaluation Can Go Off the Rails – E. Jane Davidson

Chapter 3. The Buffalo Jump: Lessons After the Fall – Gail Vallance Barrington

Chapter 4. Evaluator Self-Evaluation: When Self-Flagellation Is Not Enough – Emma Williams

PART II. ENGAGE STAKEHOLDERS

Chapter 5. That Alien Feeling: Engaging All Stakeholders in the Universe – Hallie Preskill

Chapter 6. Seeds of Failure: How the Evaluation of a West African Agricultural Scale Up Project Went Awry – Thomas Archibald

Chapter 7. I Didn’t Know I Would Be a Tightrope Walker Someday: Balancing Evaluator Responsiveness and Independence – Rakesh Mohan

Chapter 8. When National Pride Is Beyond Facts: Navigating Conflicting Stakeholder Requirements – Felix Muramutsa

PART III. BUILD EVALUATION CAPACITY

Chapter 9. Stars in Our Eyes: What Happens When Things Are Too Good to Be True – Jara Dean-Coffey

PART IV. DESCRIBE THE PROGRAM

Chapter 10. A “Failed” Logic Model: How I Learned to Connect With All Stakeholders – Chris Lovato

Chapter 11. Lost Without You: A Lesson in System Mapping and Engaging Stakeholders – Kylie Hutchinson

PART V. FOCUS THE EVALUATION DESIGN

Chapter 12. You Got to Know When to Hold ’Em: An Evaluation That Went From Bad to Worse – Robert P. Shepherd

Chapter 13. The Evaluation From Hell: When Evaluators and Clients Don’t Quite Fit – Marla Steinberg

PART VI. GATHER CREDIBLE EVIDENCE

Chapter 14. The Best Laid Plans of Mice and Evaluators: Dealing With Data Collection Surprises in the Field – Jan Noga

Chapter 15. Are You My Amigo, or My Chero? The Importance of Cultural Competence in Data Collection and Evaluation – Isaac D. Castillo

Chapter 16. OMG, Why Can’t We Get the Data? A Lesson in Managing Evaluation Expectations – Jennifer Bisgard and Mary Pat Selvaggio

Chapter 17. No, Actually, This Project Has to Stop Now: Learning When to Pull the Plug – Karen Snyder

Chapter 18. Missing in Action: How Assumptions, Language, History, and Soft Skills Influenced a Cross-Cultural Participatory Evaluation – Susan Igras

PART VII. JUSTIFY CONCLUSIONS

Chapter 19. “This Is Highly Illogical”: How a Spock Evaluator Learns That Context and Mixed Methods Are Everything – Benoît Gauthier

Chapter 20. The Ripple That Became a Splash: The Importance of Context and Why I Now Do Data Parties – Diana Tindall

Chapter 21. The Voldemort Evaluation: How I Learned to Survive Organizational Dysfunction, Confusion, and Distrust – Lisa O’Reilly

PART VIII. REPORT AND ENSURE USE

Chapter 22. The Only Way Out Is Through – Stephanie Evergreen

Conclusion

About the author

Kylie Hutchinson took her first course in evaluation in 1988 and was hooked. She won the Canadian Evaluation Society’s Student Competition in 1990, made the leap into evaluation consulting in 1998, andhas never looked back. In 2005 she was asked to deliver the Canadian Evaluation Society’s Essential Skillscourse in British Columbia which she did until 2011. This focus on training sparked a new interest inevaluation capacity-building and she enrolled in Vancouver Community College’s Instructor Development Program. By 2007 approximately 40% of her work was evaluation capacity-building. She continues to thisday providing both her own private and sponsored workshops and webinars for clients such as the American Evaluation Association, Canadian Evaluation Society, and U.S. Centers for Disease Control. Since2008, she has become increasingly interested in evaluation reporting, and has run a pre-conferenceworkshop on the topic at AEA. Kylie acknowledges her roots are firmly rooted in the muddy trenches ofevaluation practice versus academia. She has a passion for taking evaluation theory and making itaccessible for evaluators to use immediately in their practice: this is evidenced by some of the freeevaluation resources she has produced over the years including one-page reference sheets, a podcast, twovideos, an e Learning course, a mobile learning course for senior decision-makers, and an evaluationglossary mobile app. She is also the author of An Innovative Guide to Evaluation Reporting and Survive and Thrive: Three Steps to Securing Your Program Sustainability