Learning from Failure: Feel It First, Then Analyze - Natalia Romanska

Testing AI models means testing a magic box: nobody has the source code, and the behavior is neither controlled nor understood. Michael Bolton calls that strongly magical, and with him I talk about what follows from it. His principles are simple: if something matters very much, don't depend on magic, and if you use it anyway, responsibility demands that you test it and maximize testability. Simple output checks don't stand a chance against a language model, so testing becomes experiencing, exploring and experimenting, like peeling an onion. And Penn and Teller's clear plastic cups show why we fall for these models in the first place: the insight we see in their text comes from us.

Testing is supposed to prevent failures in production, yet sometimes the test itself is what breaks things. Bart Knaack calls this the dark side of testing, and we talk about test data leaking out of poorly protected environments, test systems that give attackers a look at your code, and why testing in production needs a risk assessment of its own. Bart would rather ask questions than hand out checklists, because a ticked box stops people thinking. He also tells the story of a team with 10,000 end-to-end test cases that ended up needing only 50.

Quality hunting grew out of Rik Marselis's frustration with bug hunting, and I wanted to know how it works when the goal is information about quality. Rik walks me through a session: teams of about four people, three hours of exploratory testing and a goal agreed up front by asking stakeholders what would help them sleep well at night. He explains why the report structure comes first, why the prize should stay small and why the point just before several sprints go to production suits a hunt well.

What does it take to turn a risky one-time event into a conference that testers travel across borders for? With Okan Çakmak I talk about how the Istanbul Software Testing Conference grew out of something missing: after the pandemic, the big Turkish testing events stayed online, and people lost the chance to meet, network and hug each other in person. He tells me how the team wondered whether 50 or 100 people would come, then welcomed 200, and how the "light in their eyes" of those first participants convinced them to keep going.

What happens when AI ships poor quality faster, and nobody notices because the pipeline stays green? With Michaël Pilaeten I talk about why test automation without a clear strategy is a problem that AI only accelerates, not solves. We get into the question of what separates a tester from a test automation engineer, why those two roles need each other but are not interchangeable, and how chasing 100% coverage on millions of lines of legacy code can be a waste everyone agrees to quietly.