Show Notes
Why does an AI sometimes give different answers to the exact same prompt? This episode explains temperature, the sampling setting behind that familiar slider in AI tools and APIs, and what it's controlling under the hood.
A language model generates text one token at a time, and at each step it has a probability distribution over possible next tokens. Temperature controls how that distribution is sampled. OpenAI's own API documentation describes it as a measure of how often the model outputs a less likely token, with a range from zero to two: lower values produce more focused, deterministic output, and higher values produce more random output. The episode walks through a concrete example, "I poured a cup of...", to show how a low temperature sharpens the distribution toward the most likely words, while a high temperature flattens it and gives less likely words more of a chance.
The episode explains why temperature zero is the standard choice for tasks like extracting structured data from an email, where predictable output is useful because another program has to parse it, versus a higher temperature for tasks like brainstorming names, where variation is useful. It's also direct about what temperature does not change: it doesn't add knowledge, retrain the model, or improve accuracy. A surprising answer at high temperature isn't automatically a better one, and a low temperature makes output more consistent without making an inaccurate model correct.
Because listeners often confuse temperature with a related setting, the episode also covers top P, or nucleus sampling, and OpenAI's own guidance to adjust one of these controls rather than both at once. It closes by explaining the mechanism behind an AI's inconsistency: once one token early in a response changes, the model is predicting its next token from a different context, which can send the rest of the answer down an entirely different path even from an identical prompt.
Sources & References OpenAI Help Center: Best practices for prompt engineering with the OpenAI API — https://help.openai.com/en/articles/6654000-best-practices-for-prompt-engineering-with-chatgpt OpenAI API Reference: Create a model response — https://developers.openai.com/api/reference/cli/resources/responses/methods/create OpenAI Help Center: Why am I getting different completions on Playground vs. the API? — https://help.openai.com/en/articles/6643200-why-am-i-getting-different-completions-on-playground-vs-the-api OpenAI API: Advanced usage — https://developers.openai.com/api/docs/guides/advanced-usage
Voice narration is AI-generated.
More from Learn AI in Bits
A wave of cybersecurity disclosures has made this week feel like a turning point, with people calling it "Black October." This episode treats it as a cluster of high-impact incidents, not proof that one AI-driven campaign caused them all, and separates where AI was directly involved from where the failures were conventional.
Recursive self-improvement is the idea that an AI can improve the very process it uses to improve itself, and this episode separates what's been demonstrated so far from the dramatic predictions often attached to the term. It draws a clear line between self-correction (revising one answer), automated optimization (keeping the best of several tested variants), and true recursive improvement, where the system changes the process that generates future improvements, so the next round starts with a better improver.
This week's AI News Sunday Wrap-Up covers a week where the industry moved on several fronts at once: new flagship and open models, enterprise agent governance, a serious safety warning, legal pressure, and the physical infrastructure limits behind the AI race.
Meta's Muse story is moving fast, and this episode takes a closer look at Muse Spark, the multimodal reasoning model family behind Meta's push into AI agents, including yesterday's release of an open-weight version for developers.
When a large language model generates text one token at a time, even a fast GPU hits a sequential bottleneck. This episode explains speculative decoding, an inference technique that prepares several tokens at once and checks them together, and when the extra computation pays off.