Episode 012: You Can't Prove It Worked If You Didn't Design For It — Why most program evaluation fails before the program even starts.
Learning Experience Design for ChangemakersShow Notes
Most of us find out we don't have real proof our program works right when we need it most — writing a report, walking into a renewal call, or facing a board. In this episode, Renee breaks down why that's a timing problem, not a talent problem, and walks through exactly how to fix it before your next program launches. In this episode: The one-sentence test that reveals whether you actually have evidence your program works (most people can't pass it) Why “afterthought” evaluation quietly produces thin data you can't hand to a funder or a board The reframe that changes everything: evaluation as a design commitment, not a measurement tool A real example of a program that documented a 50%+ jump in success — with a follow-up email sequence, not a research team The land-trust story that shows what happens when your objective and your evidence aren't built together How the milestones you already run (weekly calls, check-ins) can double as evaluation checkpoints without adding new work Timestamps [0:00] Opening — the one-sentence test and why evaluation is a timing problem [1:00] The pattern: evaluation treated as the last step [2:00] The line to remember — you can't retroactively collect data on a transformation you didn't design for [3:00] The reframe: evaluation as a design commitment [5:00] Thin data vs. thick data, and a 50%+ documented success story [6:00] The land-trust example — an objective with no evidence architecture [8:00] Rebuilding a program backward from one observable behavior [9:00] The milestones you already have (weekly calls, check-ins) [10:00] Milestones as evaluation checkpoints — purposeful engagement [11:00] Recap and what's next for the series
Resources mentioned Episode 6 — on designing milestones — referenced as a companion listen. [LINK TO EPISODE 6]
Call to action Send Renee your evaluation questions or the places you're getting stuck — email with “podcast share” in the subject line, or send a DM. Listener questions will help shape where the series goes next. [email protected] · www.linkedin.com/in/renee-koster
More from Learning Experience Design for Changemakers
Picture yourself taking one full week off. No email, no texts, no "sorry to bother you on vacation." Does your program keep running, or does it quietly stall until you're back? In the final episode of this three-part series, Renee Koster tackles the design mistake that holds the other two up: everything runs through you. In this episode: Why adding more volunteers can make an overloaded leader more overwhelmed, not less What the one-room schoolhouse teacher knew about running a room she couldn't stand over The shift that changes everything: capacity comes from design, not headcount The "two piles" exercise that shows exactly where you're holding your program up by hand Three moves to get work off your desk, none of which require a new hire Your one-item homework assignment (and why starting small is the point)
Picture your best intern's last day. They know which families need a reminder call, how to get the projector working, and which volunteer is great with the little ones. A few weeks after they leave, you realize almost none of it was written down. In this episode, Renee shares how to tell whether your training lives in people instead of in a place, and what to do about it before the next goodbye. In this episode: The three signs your training walks out the door: "Ask Sarah," the binder nobody trusts, and the five-place scavenger hunt What an archaeology field crew knows that most nonprofits don't: excavation is destruction, and the record is all that survives Why a certified CRM consultant is telling you new software won't fix this on its own A quick walk-out audit you can do right now (with a pen, or in your head while you drive) Three moves: capture it while they're still here, give it one home, update it once What next season looks like when new people start from where the last person left off
Every fall, a new group of volunteers walks in, and a week later the same 10 questions are back at your desk. In this episode, Renee shows why those questions aren’t a volunteer problem: they’re a map of exactly where your training needs support. Grab a pen. You’ll sort your own list and leave with three moves for training you never have to rebuild from scratch.
The Academy of Natural Sciences in Philadelphia closed its doors to the public after nearly 200 years. The day before, Renee took her family, along with more than 8,000 other people who showed up in the middle of a nor'easter. This episode is a personal one: why places like the Academy matter so much for kids, and what the rest of us can do to help them last. In this episode: Why a first visit to a museum, nature center or library is never a small thing for a kid The kids who lose the most when a public institution closes (and why public access matters) A high school volunteer who became the Academy's Manager of Animal Programs, and a young volunteer who became a veterinarian A hopeful suggestion for the Academy when the doors reopen. Not if, when. 3 quiet signs your volunteer program is wearing thin (score yourself 0–3) Why "more volunteers" won't fix it, and what will Your two-part homework, including a field trip
Have you ever had someone in your program just… go quiet? No dramatic exit, no complaint — just an unopened module and a check-in that never comes back. In this episode, Renee shares the time she was that learner, what her coach did that made it safe to tell the truth, and how you can use the same approach to hear what your quiet learners are really avoiding.