Why Databricks serverless exists, and which of your jobs should stay on classic clusters
The Databricks Data EngineerShow Notes
Open the workspace. Start a cluster. Get coffee. Come back, and sometimes it's ready. If you've been doing this a few years you can feel that wait in your shoulders, and it's the least interesting part of the story.
Before any of it, you had to answer a question you could not possibly answer: how big should this be? Start small and you pay in latency. Start big and you pay for machines that are barely breathing for most of the run. Wait, or waste, and there was never a third option.
In this episode: - Why the cluster wait was only the receipt, and what classic Databricks compute never actually solved - Why Databricks can keep compute warm when your own team never could, and what that shows up as in the rate - What "serverless" means here, jobs and notebooks, and why a serverless SQL warehouse is a different conversation - The thing a fixed cluster was quietly doing for your budget that serverless stops doing, and why the policy you're thinking of doesn't cover it - Three tests to run on your own workloads this week, plus the ones that can't move at all no matter what the math says
This episode is for Databricks data engineers who have serverless sitting in a dropdown and no real argument either way. Whether you're defending a migration in an architecture review or explaining next month's bill, you'll walk away able to say which of your jobs should move, which should stay on classic, and why.
--- Helping 18,000+ Databricks data engineers become seniors: interview like seniors, execute like seniors, think like seniors.
Follow The Databricks Data Engineer for new episodes every week.
LinkedIn: linkedin.com/in/jrlasak Newsletter: dataengineer.wiki
#DataEngineering #Databricks #DataEngineer #CareerGrowth #ApacheSpark #DeltaLake
More from The Databricks Data Engineer
The rollout is done. Every production table migrated to Unity Catalog, every one owned by a team group, and the revenue table carries a certified check mark. The platform lead posts in Slack: fully governed. Monday morning, finance replies in the thread. The revenue on the finance dashboard and the revenue in the sales deck are about three percent apart. Which one's real?
Two proposals sit on a head of data's desk, both about the nightly pipeline that feeds the finance dashboard. One swaps the full reload for incremental loads with Auto Loader: it finishes in a fraction of the time and it's cheaper every month after the first. The other keeps the full reload, puts it on a bigger job cluster and starts it an hour earlier. It costs more. Forever. She asks one question, "What happens if this breaks on quarter close?", and approves the second one.
Your nightly Databricks job ran long, so you open the Spark UI. Two hundred tasks. One hundred ninety-nine finished in about a minute. One is still running, and it's been running for forty. You double the cluster and rerun it. Same forty minutes. Nearly twice the bill.
Sam is four years into a Databricks role, up for senior this cycle. Merged pull requests on the ingestion layer. The nightly job he rescued at three in the morning when the orders table stopped landing. Then his manager asks for one call where the answer wasn't obvious, he framed it, and he lived with the result. Sam starts three different sentences. None of them finish.
You cleared the recruiter screen. You cleared the Databricks technical round, and cleared it well. Then the final round, then the email that says strong candidate, tough decision. You go back looking for the mistake and there isn't one. You were good. Somebody was just easier to picture.