KubeFM

We Broke Our EKS Cluster Autoscaler with the AL2023 Migration, with Dilshan Wijesooriya

KubeFM

Show Notes

Dilshan Wijesooriya, Senior Cloud Engineer, discusses a real incident where migrating EKS nodes to AL2023 caused the cluster autoscaler to lose AWS permissions silently. You will learn: Why AL2023 blocks pod access to instance metadata by default, breaking components that relied on node IAM roles (like cluster autoscaler, external-DNS, and AWS Load Balancer Controller)

How to implement IRSA correctly by configuring IAM roles, Kubernetes service accounts, and OIDC trust relationships, and why both AWS IAM and Kubernetes RBAC must be configured independently

The recommended migration strategy: move critical system components to IRSA before changing AMIs, test aggressively in non-production, and decouple identity changes from OS upgrades

How to audit which pods currently rely on node roles and clean up legacy IAM permissions to reduce attack surface after migration

Sponsor This episode is sponsored by LearnKube — get started on your Kubernetes journey through comprehensive online, in-person or remote training. More info Find all the links and info for this episode here: https://ku.bz/T_YPfTfDb

Interested in sponsoring an episode? Learn more.

More from KubeFM

Sep 15, 2026

Non-deterministic agents pose specific challenges for platform teams in observability, state management, governance, and trust. Mauricio (Salaboy) Salatino explains why agentic applications behave like distributed multi-agent systems. His test assigned agents to take an order, cook the pizza, deliver it, and charge the customer. One order crossed 15 containers and produced 200 traces. In this interview: Why agent frameworks can recreate monolith scaling and resource contention

Sep 8, 2026

KubeSelect is a new show that tests one bold Kubernetes hypothesis with one expert. Hosts Salman Iqbal and Bart Farrell examine the evidence and ask whether the claim holds up. In episode two, Kat Cosgrove tackles a long-running question: should teams run databases on Kubernetes? StatefulSets, persistent volumes, CSI, and database operators changed the technical answer, but they did not remove the operational risk. In this episode: How Kubernetes storage has matured and why old assumptions persist

Sep 1, 2026

Kube Select takes one bold Kubernetes hypothesis and tests it with an expert. In the first episode, Salman Iqbal and Bart Farrell ask whether most organizations have an observability problem or a decision-making problem. Henrik Rexed, Senior Staff Engineer at Dynatrace, challenges the original claim. Teams can collect metrics, logs, and traces, but that telemetry needs system relationships, ownership, and deployment history to support a confident decision. In this episode: Why telemetry without context slows incident response

Aug 25, 2026

Kube Signals starts where the keynote ends: with the trends that platform teams will have to operationalize next. In this special episode, Brian Teller speaks with Saiyam Pathak about his KubeCon India keynote and the shift from developer platforms to AI factories. They examine what GPU scarcity, shared accelerators, and AI workloads mean after the conference slides meet real infrastructure. In this interview: Why GPU infrastructure is becoming a platform-engineering concern

Aug 19, 2026

At enterprise scale, a deployment pipeline that runs Helm upgrades directly against Kubernetes hides drift, mixes configuration with CI logic, and makes the last pipeline run the source of truth. Elad Cohen explains how WSC Sports moved from Azure DevOps to GitHub Actions and redesigned delivery around Git and Argo CD. The resulting platform separates builds from deployments, keeps service configuration in values files, and continuously reconciles clusters. In this interview: Why CI should change Git instead of the cluster

See all episodes →