HOAI 4: Is AI Deep Research Worth the Wait? - Four AI Deep Research Tools Get One Hard Question and a Stopwatch Video
Hands-On AI (Video)Summary
A recent exploration into the capabilities of AI research modes reveals significant differences in performance among various systems. The discussion centers on the effectiveness of deep research tools, which allow users to pose complex questions and receive detailed reports with sourced information. The host, Micah Sargent, conducts a comparative analysis of four AI systems—ChatGPT, Claude, Perplexity, and Gemini—by posing a challenging question regarding the potential replacement of a gas furnace with a heat pump for a home in Sacramento.
The results show that while the control group, which used standard web search, provided a quick and accurate response in two minutes and 34 seconds, the deep research tools varied in both time and accuracy. ChatGPT's deep research mode took four minutes but produced a report with several inaccuracies, while Claude, despite taking five minutes and citing numerous sources, also required significant cleanup. Perplexity and Gemini, which took longer, had mixed results regarding the accuracy of their citations and the relevance of their sources.
Ultimately, the findings suggest that spending more time on research does not necessarily guarantee better results. The control group's quick response was often more reliable than the deeper analyses provided by the other tools. The host emphasizes the importance of verifying AI-generated reports by checking citations and ensuring that the information is current and relevant to the user's specific context. This exercise serves as a reminder of the need for critical evaluation when utilizing AI for research purposes.
Show Notes
The big AI apps now have research modes that make you wait for a report full of sources. Is the wait worth it? Mikah Sargent gives ChatGPT, Gemini, Claude, and Perplexity the same complicated question, plus ChatGPT with regular web search as a control, and compares how long they take, where they look, what they miss, and how much cleanup each report needs. Then he checks every answer against a rebate and tax-credit answer key built from official sources.
Host: Mikah Sargent
Download or subscribe to Hands-On AI at https://twit.tv/shows/hands-on-ai
Want access to the ad-free audio and video and exclusive features? Become a member of Club TWiT today! https://twit.tv/clubtwit
Club TWiT members can discuss this episode and leave feedback in the Club TWiT Discord.
Sponsor: canary.tools/twit - use code: TWIT
More from Hands-On AI (Video)
What happens when AI stops talking and starts doing? Mikah Sargent puts Meta's Muse agent to work on simple jobs, hard ones, and a couple of weird ones: a weekly news brief, a day trip, a shopping cart that stops for his approval, a web page it builds itself, and a podcast about whether a hot dog is a sandwich. Along the way, he explains what makes an agent different from a chatbot, what Muse can and can't touch, and the settings to check before you hand any AI the keys.
What's actually inside an AI model? Mikah Sargent opens up the file from last week's episode and finds tokens, billions of numbers, and one surprisingly simple job: guess the next piece of text. Along the way, he shows why the same question can get a different answer every time, where all those numbers came from, and why a chatbot can sound completely sure of itself and still be wrong.
Your next chatbot doesn't need the cloud. Mikah Sargent installs a free app, downloads an AI model, turns off the Wi-Fi, and keeps right on chatting, all on an ordinary computer. Along the way, he explains how to read a model listing, including what the size and "Q4" labels mean, so you can pick a model that runs well on your own machine. Then the local model goes head-to-head with a cloud assistant to see where it holds up and where it falls short.
Hands-On AI is the show that turns AI confusion into AI confidence. In this introductory episode, Mikah Sargent lays out what the show is and isn't: short, focused episodes — most 10 to 15 minutes — that take one question, one tool, or one technique at a time. Some weeks go under the hood on what a large language model is actually doing. Some weeks get practical: the AI already in your pocket, or a model running entirely on your own computer. And some weeks are honest verdicts on the tools everyone's talking about.
Qwen, Muse Glimmer, Ollama, llama.cpp, quantization, mixture-of-experts, RAG, tokens per second. On Hands-On AI, those aren't buzzwords — Mikah Sargent will explain exactly what they mean, then put them to work on hardware you already own. No cloud, no account, no data, leaving your desk.