How Photon actually makes your Databricks queries faster (and when it silently doesn't)
The Databricks Data EngineerShow Notes
Two engineers run the same SQL on the same Delta table. Same data, same cluster size, copy-pasted code. Alex goes to make a coffee and comes back to a query still running. Sam's is done before they finish reading the first Slack message. The only difference is one checkbox on the cluster called Photon.
Most Databricks data engineers have that box ticked, pay a premium for it on every DBU, and could not explain in plain English what it actually does. "Photon makes it faster" isn't an answer - it's just saying the box does what the box does. And there's one thing it quietly stops doing while you keep paying for it.
In this episode: - The single architectural change underneath Photon, explained without one line of code or config - A mental model that lets you explain Photon to a PM in 60 seconds flat - Why Photon flies on scans, filters, joins, and aggregations but does nothing for certain custom code - The silent fallback that charges you the fast-engine premium for slow-engine work, with nothing on screen to warn you - The one query to open in the query profile this week, and the single question to ask of it
This episode is for Databricks data engineers who run Photon every day and treat it as a speed toggle they never think about. Whether you're a mid-level engineer who clicks the box and moves on, or a senior who's never checked whether the premium is paying off, you'll walk away knowing exactly how to tell when Photon is doing the work you're being billed for - and when it isn't.
--- Helping 18,000+ Databricks data engineers become seniors: interview like seniors, execute like seniors, think like seniors.
Follow The Databricks Data Engineer for new episodes every Monday, Wednesday, and Friday.
LinkedIn: linkedin.com/in/jrlasak Newsletter: dataengineer.wiki
#DataEngineering #Databricks #DataEngineer #CareerGrowth #ApacheSpark #DeltaLake
More from The Databricks Data Engineer
The rollout is done. Every production table migrated to Unity Catalog, every one owned by a team group, and the revenue table carries a certified check mark. The platform lead posts in Slack: fully governed. Monday morning, finance replies in the thread. The revenue on the finance dashboard and the revenue in the sales deck are about three percent apart. Which one's real?
Two proposals sit on a head of data's desk, both about the nightly pipeline that feeds the finance dashboard. One swaps the full reload for incremental loads with Auto Loader: it finishes in a fraction of the time and it's cheaper every month after the first. The other keeps the full reload, puts it on a bigger job cluster and starts it an hour earlier. It costs more. Forever. She asks one question, "What happens if this breaks on quarter close?", and approves the second one.
Your nightly Databricks job ran long, so you open the Spark UI. Two hundred tasks. One hundred ninety-nine finished in about a minute. One is still running, and it's been running for forty. You double the cluster and rerun it. Same forty minutes. Nearly twice the bill.
Sam is four years into a Databricks role, up for senior this cycle. Merged pull requests on the ingestion layer. The nightly job he rescued at three in the morning when the orders table stopped landing. Then his manager asks for one call where the answer wasn't obvious, he framed it, and he lived with the result. Sam starts three different sentences. None of them finish.
Open the workspace. Start a cluster. Get coffee. Come back, and sometimes it's ready. If you've been doing this a few years you can feel that wait in your shoulders, and it's the least interesting part of the story.