
E106: Boeing 737 MAX Disaster: When High-Authority Systems Go "Strong, Silent, and Wrong"
Safe As PodcastShow Notes
What happens when a high-authority automated system operate with an incorrect model of reality?This question is explored via Dekker & Woods’ (2024) article on automated systems that are Strong, Silent, and Wrong.We explore the breakdown of human-automation coordination, mode confusion, and automation surprises, using the Boeing 737 MAX MCAS accidents as a central example. Learn how complex system design must prioritize observability, redirectability, and graceful failure.Ref: Dekker, S. W., & Woods, D. D. (2024). Wrong, strong, and silent: What happens when automated systems with high autonomy and high authority misbehave? Journal of Cognitive Engineering and Decision Making, 18(4), 339–347.Resources & Links:Website & Articles: https://safetyinsights.orgSupport the channel: https://buymeacoffee.com/benhutchinson
More from Safe As Podcast
Are higher-energy hazard interactions more likely to seriously injure or kill people?In this ep, we unpack the empirical link between energy exposure and SIFs - where over 500 injury reports were analysed, and injury energy magnitude was estimated using leading information.Key concepts were energy magnitude - being the amount of energy present; and energy intensity - how concentrated that energy is over the contact area. E.g. a sharp object can create greater harm than a blunt object carrying the same energy.Study Reference:Hallowell, M. R., et al. "Energy-based safety risk assessment: does magnitude and intensity of energy predict injury severity?" Construction Management and Economics.
Can AI help us identify critical safety blind-spots before disaster?Part 2 interview with Professor Robin Dillon-Merrill (Georgetown University, McDonough School of Business), who unpacks her 20+ years of research on decision-making under risk, outcome bias, and how organizations misinterpret near misses.We discuss:a) why near-miss biases shape human decision-making & how we can build robust systems to counter them catch them b) why technical experts are uniquely positioned to simulate counterfactuals ('what could have happened, but didn't')c) how well-intended safety messaging becomes background noise without institutional actiond) how cutting-edge AI & machine learning can triage gigantic incident datasets before catastrophe.Plus, Robin discusses real-world safety challenges with autonomous vehicles, such as Waymo vehicles illegally passing school buses in Texas.Resources & Links:Website & Articles: https://safetyinsights.orgSupport the channel: https://buymeacoffee.com/benhutchinson
Today we unpack a study tracking 1,422 procedural steps across 40 petrochemical operations. Using their Skip-Order-Action (SOA) Framework, they explored how frontline personnel navigate written instructions in practice: revealing that roughly 1/3 of steps depart from prescribed methods even when tasks are completed successfully. We explore how: a. the SOA framework functions b. how workers using paper instructions followed fewer steps as the task progressed (e.g. dropping from ~75% adherence to 55%)c. what these findings mean for safety professionals advancing a Safety-II / HOP approachReference: Ashraf, A. M., Peres, S. C., & Sasangohar, F. (2025). Beyond Compliance: Investigating Work-as-Done in Procedural Work with the Systematic Skip-Order-Action (SOA) Framework. Resources & Links:Website & Articles: https://safetyinsights.orgSupport the channel: https://buymeacoffee.com/benhutchinson
Can fatigue cause visual blindness in plant operators and drivers?In this video, I review a 2019 study from Li et al. (Automation in Construction) who used wearable eye-tracking tech to measure how prolonged excavator operation / time-on-task fatigue impacts hazard detection and visual attention.Namely, whether the operators notice workers walking behind the excavator over a shift.Key findings:a. Hazard miss rates increased over 40% after 60 mins. Overall detection rates dropped to 70% at 36 minutes and down to 60% by the 60-minute mark.b. Operators took significantly longer to recognize peripheral hazards as fatigue increased.c. Visual attention on the excavator side mirrors and rearview mirrors decreased over the task. Operators stopped scanning peripheral blind spots and focused almost exclusively on the central display directly in front of them.
Do we routinely mistake good luck with systemic resilience – until something blows up? Today I speak with Professor Robin Dillon-Merrill (Georgetown University, McDonough School of Business), who breaks down her 20+ years of research on decision-making under risk, outcome bias, and how organizations misinterpret near misses. We discuss: • The differences between resilient vs vulnerable near misses: How the perceived ‘margin of safety’ desensitizes organizations to risk when luck is the primary reason failure was averted. • Managing outcome bias: Why leaders who experience lucky successes are possibly judged as more competent and promotable than those who face bad luck, even when their underlying decision quality is identical. • Counterfactual thinking: How processes used by NASA facilitate teams to evaluate alternative outcomes and decouple decision quality from outcomes. • Gathering info: Why decision makers consistently stop searching for critical warning signs long before they have sufficient data. • Why ‘speaking up’ campaigns fail: Safety reporting programs may be ineffective for recognising an issue since the issue isn’t recognised as danger in the first place.