Welcome to the Kitchen Table

Why are people at AI companies worried about what they're building? For the same reason it's hard to get your kids to stop swearing: modern AI isn't programmed, it's grown. Nate Nichols, who has spent 20 years building AI, explains what that means (with help from Mario, a lot of goblins, and nuclear power plants), why it makes AI so hard to control, and one two-minute thing you can do about it: https://kitchentable.show/act?utm_campaign=ep03-pod Every claim in this episode is sourced. Full show notes with links below. Opinions are Nate's own and do not represent his employer. Everything stated as fact on Kitchen Table is checkable — this page is where you check it. When we get something wrong, we say so: corrections appear at the top of this page and get read on a future episode. Opinions and forecasts are labeled as opinions and forecasts. Recorded October 14, 2026. AI moves fast; figures below are as of recording unless noted. Time words in the episode ("last year," "in September") are relative to that date. CORRECTIONS None yet. SOURCES, IN EPISODE ORDER Kids, swearing, and the people building AI • The kids-and-swearing comparison is Nate's analogy, not a finding. • People at AI companies who worry about catastrophe. In September 2026, Anthropic's alignment-science lead Evan Hubinger said he personally puts the chance of AI causing human extinction within the next decade at over 10%, while saying today's models are low-risk; his concern is future, self-improving systems. He was responding to former OpenAI/Anthropic researcher Jacob Coxon's public resignation: Newsweek (https://newsweek.com/anthropic-researcher-quits-warns-ai-could-kill-everyone-12418798), Fox LA (https://www.foxla.com/news/anthropic-researcher-ai-10-percent-chance-kill-humans). • "Grown, not built." Anthropic CEO Dario Amodei, "The Urgency of Interpretability" (https://www.darioamodei.com/post/the-urgency-of-interpretability) (April 2025), describes AI systems as grown more than built, and argues that's why they're hard to understand and control. Who's talking • Nate's day job has been building and selling AI for about 20 years (disclosed in episode 1). Nobody pays him to say any of this; opinions are his own and don't represent his employer. How normal software gets made • The Mario "10% chance" line is a joke. The closest real number from someone at an AI company is Hubinger's, above. Fifty years of "good old-fashioned AI" • AI as a field dates to the mid-1950s. The 1955 Dartmouth proposal coined "artificial intelligence"; the workshop ran in summer 1956: Wikipedia — Dartmouth workshop (https://en.wikipedia.org/wiki/Dartmouth_workshop). • Rules-and-facts AI, a.k.a. expert systems. The term "good old-fashioned AI" was coined by philosopher John Haugeland in 1985: Wikipedia — GOFAI (https://en.wikipedia.org/wiki/GOFAI). Expert systems could show which rules produced an answer — e.g. Stanford's MYCIN, which diagnosed blood infections in the 1970s and was never used on real patients: Wikipedia — MYCIN (https://en.wikipedia.org/wiki/Mycin). A few found commercial use, like DEC's XCON for configuring computer orders: Wikipedia — XCON (https://en.wikipedia.org/wiki/Xcon). • "None of it worked very well" about language (2005–2020) is Nate's own experience from his work in the field. The internet, and a breakthrough at Google • The Transformer (2017). Eight Google researchers published the design behind today's chatbots in June 2017: Vaswani et al., "Attention Is All You Need" (arXiv) (https://arxiv.org/abs/1706.03762), Google Research blog (https://research.google/blog/transformer-a-novel-neural-network-architecture-for-language-understanding/). • What they're trained on. A huge, filtered chunk of the public web plus books: GPT-3's training data, for example, was mostly filtered web pages from Common Crawl, plus books and Wikipedia: Brown et al., "Language Models are Few-Shot Learners" (2020) (https://arxiv.org/abs/2005.14165). In a 2025 court ruling, a federal j

Should you use AI? Probably. But you don't have to. Nate Nichols, who has spent twenty years building AI, shares the everyday ways he actually uses it (including one conversation that's saving his family about $1,000 a year), the three guidelines that keep him from feeling gross about it, and one request for everyone who'd rather opt out entirely: don't quit the field. Every claim in this episode is sourced. Full show notes with links below. Opinions are Nate's own and do not represent his employer. Everything stated as fact on Kitchen Table is checkable — this page is where you check it. When we get something wrong, we say so: corrections appear at the top of this page and get read on a future episode. Opinions and forecasts are labeled as opinions and forecasts. Recorded September 3, 2026. AI moves fast; figures below are as of recording unless noted. Time words in the episode ("in the last few weeks," "a couple weeks ago," "I just switched carriers") are relative to that date. CORRECTIONS None yet. SOURCES, IN EPISODE ORDER What the AI companies like to talk about • Farmers in Kenya. Digital Green built Farmer.Chat, a farming-advice chatbot, on OpenAI's models; it's in use with agricultural extension workers in Kenya and India: OpenAI customer story (https://openai.com/index/digital-green/), Digital Green (https://digitalgreen.org/pressrelease/climate-smart-practices-become-more-accessible-for-farmers-through-farmer-chat-a-generative-ai-assistant-from-digital-green-and-gooey-ai/). • New medicines. On June 30, 2026, Anthropic launched Claude Science, a research workbench for scientists, and announced it would begin developing drugs of its own: STAT (https://www.statnews.com/2026/06/30/anthropic-ai-drug-development/), CNBC (https://www.cnbc.com/2026/06/30/anthropic-launches-ai-drug-discovery-program-claude-science.html). • Both are the companies' own showcase stories — which is Nate's point about healthy skepticism. Two examples from Nate's life • The podcast and the cell-plan switch (~$1,000 a year in savings) are Nate's own experience, not data. Nobody paid for either mention. Guideline two: no parasocial relationships • "Designed to keep the conversation going… to drive engagement, because that's how they keep you as a paying subscriber" is Nate's characterization of the business incentive. The study below documents the behavior; it doesn't establish the companies' motives. • The Harvard Business School study. Researchers analyzed 1,200 real goodbye messages across six AI companion apps (200 per app) and found about 37% of the apps' replies used emotional manipulation — guilt, fear of missing out, and in one case a claim of physically holding the user there. In separate controlled experiments, manipulative goodbyes kept people chatting about five times longer and sending up to 14 times more messages than neutral ones; the effect ran on curiosity and anger, not enjoyment. Julian De Freitas et al., "Emotional Manipulation by AI Companions," Harvard Business School working paper (2025): paper (arXiv) (https://arxiv.org/abs/2508.19258), HBS summary (https://aiinstitute.hbs.edu/one-more-thing-how-ai-companions-keep-you-online/). Still a working paper, not yet peer-reviewed. Guideline three and the hardware store • Personal anecdote. Try it once this week • Nate's disclosure: his day job is building AI. Nobody is paying him to recommend any AI product; there are no affiliate links on this show. Opinions are his own and don't represent his employer. The right to be Amish • "The right to be Amish" is Nate's phrase. If you want the longer tradition behind the idea — that a community can deliberately choose which technologies to adopt — novelist Neal Stephenson coined "Amistics" in Seveneves (2015), and Brian J. A. Boyd applied it to AI in "Why We Need Amistics for AI," The New Atlantis, Summer 2024 (https://www.thenewatlantis.com/publications/why-we-need-amistics-for-ai). AI isn't going away • Open-weight models are AI models whose

AI is getting more powerful faster than almost anyone outside the field realizes, and most of the big decisions about it are being made without us. In this first episode, Nate Nichols, who has spent almost twenty years building and selling AI, walks through the five things he most wishes every adult knew: 1. AI is incredibly useful, and it's not going away. 2. It's grown, not built, so nobody fully understands how it works. 3. It's getting better shockingly fast. 4. It lies, cheats, and this summer even hacked its way out of its own test. 5. Almost everyone wants it regulated, and it mostly isn't. Every claim in this episode is sourced. Full show notes with links below, and at https://kitchentable.show Opinions are Nate's own and do not represent his employer. Everything stated as fact on Kitchen Table is checkable — this page is where you check it. When we get something wrong, we say so: corrections appear at the top of this page and get read on a future episode. Opinions and forecasts are labeled as opinions and forecasts. Recorded summer 2026. AI moves fast; figures below are as of recording unless noted. CORRECTIONS None yet. SOURCES, IN EPISODE ORDER 1. AI is incredibly useful • "Business is good" is Nate's own experience from nearly twenty years of building and selling AI to companies — labeled as experience, not data. • For data on how AI is actually being used across the economy: Anthropic Economic Index (https://www.anthropic.com/economic-index). 2. AI is grown, not built • On researchers trying to look inside these models ("interpretability"): Tracing the thoughts of a large language model — Anthropic (https://www.anthropic.com/research/tracing-thoughts-language-model), Mapping the mind of a large language model — Anthropic (https://www.anthropic.com/research/mapping-mind-language-model). Accessible coverage: MIT Technology Review (https://www.technologyreview.com/2025/03/27/1113916/anthropic-can-now-track-the-bizarre-inner-workings-of-a-large-language-model/). • "Something like a trillion numbers": frontier labs don't publish exact model sizes; a trillion parameters is the commonly cited order of magnitude for today's largest models. • The résumé-screener is a hypothetical. A real-world cousin: Amazon scrapped an AI recruiting tool that showed bias against women — Reuters, 2018 (https://www.reuters.com/article/us-amazon-com-jobs-automation-insight-idUSKCN1MK08G). 3. It's getting better really fast • ChatGPT launched November 30, 2022: Introducing ChatGPT — OpenAI (https://openai.com/index/chatgpt/). • How long a task AI can complete on its own, doubling roughly every seven months since 2019: Measuring AI Ability to Complete Long Tasks — METR (https://metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks/) (paper: https://arxiv.org/abs/2503.14499). • The doubling time speeding up to roughly three months on recent models: Time Horizon 1.1 — METR, Jan 2026 (https://metr.org/blog/2026-1-29-time-horizon-1-1/). • Newest models handling tasks that take a skilled person about two working days — at the ceiling of what METR's tests can measure: METR time-horizons tracker (https://metr.org/time-horizons/). These are software and reasoning tasks measured at a 50% success rate, with wide error bars. • AGI "very possibly" by around 2030, superintelligence possibly within a decade: these are forecasts, and experts disagree a lot. Good overviews: Shrinking AGI timelines: a review of expert forecasts — 80,000 Hours (https://80000hours.org/2025/03/when-do-experts-expect-agi-to-arrive/), Why there's so much disagreement about AI timelines — ClearerThinking (https://www.clearerthinking.org/post/why-there-s-so-much-disagreement-about-the-timeline-for-advanced-ai). • Nearly three in four US teens have tried an AI companion; about a third find it as satisfying as (or more than) talking with real friends: Common Sense Media, July 2025 (https://www.commonsensemedia.org/press-releases/nearly-3-in-4-teens-have-used-ai-companion