Diagnostic Testing: Do the stats tell you what you need to know?

A famous Harvard researcher. A blockbuster finding about honesty. Suspicious data, failed replications, and a $25 million lawsuit. Welcome to one of the strangest research misconduct scandals in recent memory. In the first of two episodes about the Francesca Gino case, we investigate a famous experiment claiming that signing an honesty pledge at the top of a form makes people less likely to cheat. We follow the trail from a failed replication to some extraordinary detective work involving an Excel spreadsheet—and conduct a little statistical sleuthing of our own. Along the way, we discover how suspicious changes to a handful of observations can alter the story a scientific study tells. Statistical topics Statistical sleuthing Data integrity Data management Replication Sensitivity analysis Effect sizes Statistical significance Reproducibility Spreadsheets Conceptual replication Direct replication CalcChain Methodologic Morals “There's cleaning your data, and then there's laundering your data.” “Good data record keeping is still sexy. Version control, baby.”

Can you spot problems in a scientific study just by reading the published paper? We put our statistical sleuthing skills to the test with a famous study on procrastination and deadlines that was retracted more than 20 years after it was published. But were there clues hidden in plain sight in the published results—and would we have noticed anything suspicious at the time? We put on our data-detective hats and find a few statistical clues of our own before turning to Data Colada’s investigation of the original data, where the story gets even stranger. Along the way, we hunt for missing participants, question suspiciously enormous effects, meet some unlikely data twins, and discover why sometimes the best statistical tool is simply asking: Do these data behave like real people?

What happens when you discover that your published paper is wrong? Does retracting it ruin your reputation—or could owning the mistake actually make things better? In this episode, we meet the inaugural winners of the Ctrl-Z Awards, which recognize scientists for correcting or retracting their own work. Behavioral ecologist Kate Laskowski takes us inside a scientific detective story involving suspicious spider data, three retracted papers, and the fear that going public might tank her career. Then computational social scientist Alyssa Smith tells us how one small error in a figure led her to retract the paper at the center of her PhD and job talk. Along the way, we talk about data forensics, the stigma around retractions, and why self-correcting scientists are the heart of science.

Statistical power is one of the most important ideas in statistics—and one of the most misunderstood. In this episode, we explain what statistical power really means, why researchers need to think about it before they collect their data, and what goes wrong when they calculate “post hoc power” after the study is over. Along the way, we hunt for UFOs, test Kristin’s psychic abilities, explore the ethics of underpowered studies, and discover why post hoc power is really just a p-value wearing a fake mustache. Then statistician and powerlifter Andrew Althouse joins us for the inside story of a years-long scientific feud over post hoc power, where we discuss what happens when being wrong about statistics turns into doubling down, angry letters to the editor, and debates about how science corrects itself.

Can weed protect your brain as you get older? We unpack a study linking lifetime cannabis use to larger brain volumes and better cognitive test scores in older adults—and investigate how those findings became headlines urging readers to “pass the pot to grandma.” Along the way, we explore collapsing categorical variables, confounding, clinically versus statistically significant results, multiple testing, and how a press release subtly changed a story. We also meet permanently stoned mice, learn about Carl Sagan’s surprising connection to cannabis research, and discover how a single missing word can change the entire meaning of a study. Statistical topics Categorical variables Confounding Effect size False discovery rate Healthy volunteer bias Measurement error Multiple testing Observational studies Recoding variables Science communication Statistical vs practical significance