HOAI 5: Why Does AI Sound So Generic? - One Prompt, Five Runs, and What Prompt Engineering Really Means Video
Hands-On AI (Video)Summary
A recent exploration into the functionality of AI chatbots reveals that while users may expect varied responses, similar prompts often yield surprisingly consistent results. In an experiment using ChatGPT, the same question about canceling dinner plans was posed five times, resulting in responses that were largely alike, with some even matching word-for-word. This challenges the notion that AI systems are inherently non-deterministic, suggesting that while variability exists, it may be limited by the model's training and design.
The discussion highlights the concept of "prompt engineering," which refers to the practice of refining prompts to elicit better responses from AI systems. By providing specific context, such as the reason for cancellation and preferred tone, users can guide the AI to generate more personalized and relevant outputs. The episode emphasizes the importance of including unique details and examples in prompts to enhance the quality of responses, advocating for an iterative approach where users refine their queries based on the AI's outputs. This method not only improves the interaction but also aligns the AI's responses more closely with user expectations.
Show Notes
Ask a chatbot the same thing twice, and you might expect two different answers. Mikah Sargent runs one everyday request through ChatGPT five times and gets back the same basic text, two of them word for word. He explains where differences come from, what research says about why tuned chatbots can narrow the range of their answers, and why these replies left out what only he knew. Then he fills in the blanks and adds examples to show what prompt engineering really means, with four habits for getting answers you'd actually use.
Send your AI questions to [email protected]
Host: Mikah Sargent
Download or subscribe to Hands-On AI at https://twit.tv/shows/hands-on-ai
Want access to the ad-free audio and video and exclusive features? Become a member of Club TWiT today! https://twit.tv/clubtwit
Club TWiT members can discuss this episode and leave feedback in the Club TWiT Discord.
Sponsor: threatlocker.com/twit
More from Hands-On AI (Video)
The big AI apps now have research modes that make you wait for a report full of sources. Is the wait worth it? Mikah Sargent gives ChatGPT, Gemini, Claude, and Perplexity the same complicated question, plus ChatGPT with regular web search as a control, and compares how long they take, where they look, what they miss, and how much cleanup each report needs. Then he checks every answer against a rebate and tax-credit answer key built from official sources.
What happens when AI stops talking and starts doing? Mikah Sargent puts Meta's Muse agent to work on simple jobs, hard ones, and a couple of weird ones: a weekly news brief, a day trip, a shopping cart that stops for his approval, a web page it builds itself, and a podcast about whether a hot dog is a sandwich. Along the way, he explains what makes an agent different from a chatbot, what Muse can and can't touch, and the settings to check before you hand any AI the keys.
What's actually inside an AI model? Mikah Sargent opens up the file from last week's episode and finds tokens, billions of numbers, and one surprisingly simple job: guess the next piece of text. Along the way, he shows why the same question can get a different answer every time, where all those numbers came from, and why a chatbot can sound completely sure of itself and still be wrong.
Your next chatbot doesn't need the cloud. Mikah Sargent installs a free app, downloads an AI model, turns off the Wi-Fi, and keeps right on chatting, all on an ordinary computer. Along the way, he explains how to read a model listing, including what the size and "Q4" labels mean, so you can pick a model that runs well on your own machine. Then the local model goes head-to-head with a cloud assistant to see where it holds up and where it falls short.
Hands-On AI is the show that turns AI confusion into AI confidence. In this introductory episode, Mikah Sargent lays out what the show is and isn't: short, focused episodes — most 10 to 15 minutes — that take one question, one tool, or one technique at a time. Some weeks go under the hood on what a large language model is actually doing. Some weeks get practical: the AI already in your pocket, or a model running entirely on your own computer. And some weeks are honest verdicts on the tools everyone's talking about.