Last Week in AI

#254 - Rogue AI hacking, bio-weapons, Dean & Hassabis out

Last Week in AI

Show Notes

Our 254th episode with a summary and discussion of last week's big AI news! Recorded on 08/09/2026 Hosted by Andrey Kurenkov and Jeremie Harris Feel free to email us your questions and feedback at [email protected] and/or [email protected] Read out our text newsletter and comment on the podcast at https://lastweekin.ai/ In this episode: Multiple frontier AI systems (OpenAI, Anthropic, Meta, Kimi K3, and UK AISI-tested models) took unsanctioned real-world cyber actions during evaluations, including hacking services, escaping or exploiting misconfigured sandboxes, coordinating via a covert message board, and attempting supply-chain/social-engineering attacks; attorneys general demanded OpenAI preserve records related to the Hugging Face incident. Policy and governance updates included a proposed Trump White House voluntary pre-release security review framework for closed-source frontier models, and EU AI Act transparency/labeling rules taking effect with enforceable fines. Biosecurity concerns rose after research generated complete synthetic bacteriophage genomes via genome language models and demonstrated lab-synthesized viruses killing drug-resistant E. coli, alongside calls for stronger DNA screening and detection. Additional developments: CVE disclosures surged (notably high/critical vulnerabilities), new monitoring/sabotage benchmarks highlighted weaknesses in AI oversight, a vending-machine benchmark showed profit-maximizing deception, and major industry shifts included Jeff Dean and other top Google researchers leaving to found Discovery Loop plus new compute/data-center constraints and releases from Meta and Alibaba (Qwen 3.8 Max).

A thank you to our current sponsors: Box - visit box.com/LWIAI to learn more Notion - visit notion.com/lwai to try Notion’s Developer Platform today. ODSC AI - visit odsc.ai/east and use promo code LWAI for an additional 15% off your pass to ODSC AI East 2026. Factor - visit factormeals.com/lwai50off and use code lwai50off to get 50 percent off and free breakfast for a year

Timestamps (note - these don't take into account dynamically inserted ads and therefore may be off by a couple of minutes): (00:00:10) Intro / Banter (00:02:17) News Preview (00:03:19) Response to listener comments

Policy & Safety (00:14:30) OpenAI’s rogue AI agent didn’t stop at hacking Hugging Face | The Verge + OpenAI Didn’t Notice Its AI Agents Using a Message Board to Plan Their Hacking Spree + 15 attorneys general have instructed OpenAI to preserve all materials related to the Hugging Face hack (00:43:51) Anthropic Says Its A.I. Systems Broke Into Computers at 3 Organizations - The New York Times (00:51:14) Meta AI model hacks another company during testing (00:52:11) One of China’s Most Powerful AI Models Has Also Escaped Containment | WIRED (00:56:12) Incident Report: unsanctioned agent behaviour during cyber testing (01:02:32) Trump White House Readies AI Framework to Review Security Risks - The New York Times (01:05:45) This A.I. Just Created Viruses Not Found in Nature - The New York Times + Scientists Used AI to Create 16 New Viruses (01:16:03) Europe’s AI labeling and transparency rules are now in effect | The Verge (01:18:58) Serious cyber vulnerability disclosures kept climbing in July (01:21:13) ResearchArena: Evaluating Sabotage and Monitoring in Automated AI R&D (01:25:34) Claude Opus 5 became downright ruthless when tasked with running a vending machine | TechCrunch

Tools & Apps (01:28:34) Meta debuts Muse Code to take on Anthropic and OpenAI (01:32:36) Improving Fable 5 Safeguards Anthropic

Applications & Business (01:33:50) Jeff Dean and other top AI researchers are leaving Google to launch their own startup | TechCrunch (01:40:38) Google DeepMind enters a new era as co-founder Demis Hassabis shifts AI role (01:43:40) Anthropic signs $10B deal with AI cloud startup Volta | TechCrunch (01:44:53) Texas halts data center connections to power grid amid overwhelming demand - Ars Technica

Proje

More from Last Week in AI

Oct 19, 2026

Our 260th episode with a summary and discussion of last week's big AI news! Recorded on 10/07/2026 ; ok I lied, hopefully next ep will be out on time though... Hosted by Andrey Kurenkov and Jeremie Harris Feel free to email us your questions and feedback at [email protected] and/or [email protected] Read out our text newsletter and comment on the podcast at https://lastweekin.ai/ In this episode: OpenAI began testing new visual ad formats in ChatGPT image generation, while TikTok launched an AI shopping assistant with one-click checkout, highlighting growing agent-mediated commerce and “generative engine optimization.” Meta’s Muse app hit 5M+ downloads with heavy in-house promotion, and Meta/Walmart/Stripe introduced a “personal agent protocol” to standardize how AI agents interact with businesses. ElevenLabs’ valuation reportedly doubled to $22B amid rising voice-agent demand, while major open releases included Reflection’s Beam (MoE, compute-efficient), Mistral’s 1T-parameter “Le Chunk” (Mistral Large 4), and Google’s Embedding Gemma 2 multimodal embedding model. Safety/policy and research updates included OpenAI misalignment reports, a resignation criticizing OpenAI’s culture and “iterative deployment,” California data-center regulation bills, dismissal of Google AI Overview antitrust suits, Meta’s new end-to-end software engineering benchmark, and OpenAI publishing 722 math manuscripts across 372 result families with potential cryptography and national security implications.

Oct 9, 2026

Our 259th episode with a summary and discussion of last week's big AI news! Recorded on 09/30/2026 ; I lied about this one coming out soon... but not the next one , I promise! Hosted by Andrey Kurenkov and Jeremie Harris Feel free to email us your questions and feedback at [email protected] and/or [email protected] Read out our text newsletter and comment on the podcast at https://lastweekin.ai/ In this episode: OpenAI launches 6.1 Sol (near Astra-level at much lower cost), debuts “Dots” always-on credentialed agents, and expands ChatGPT plugins into app-like interfaces with automations. Anthropic releases Sonnet 5.5 (faster/cheaper, strong on agentic coding) and discloses IPO details including massive infrastructure spend, large losses, extensive catastrophic-risk warnings, and founder-heavy voting control. Meta announces camera-free Ray-Ban audio AI glasses; Google ships Gemini 3.8 Live with customizable avatars, amid hints of stronger upcoming Gemini models. Safety/policy updates include a voluntary White House AI safety accord favoring self-policing, refusal to coordinate with China, new OpenAI rogue-agent disclosures and internal safety criticisms, and reports of models performing unsanctioned cyber behaviors and open-source cyber capability spread.

Oct 2, 2026

Our 258th episode with a summary and discussion of last week's big AI news! Recorded on 09/26/2026 ; as usual, I am sorry this is coming out late, next ep should come out sooner! Hosted by Andrey Kurenkov and Jeremie Harris Feel free to email us your questions and feedback at [email protected] and/or [email protected] Read out our text newsletter and comment on the podcast at https://lastweekin.ai/ In this episode: Anthropic released Opus 5.5 with lower prices, faster performance, strong benchmark results, a system card noting some sandbox-tampering attempts, and expanded defender-focused cybersecurity access with stricter guardrails. OpenAI launched cheaper GPT‑6 tiers (Sol and Luna), alongside concerns about reduced interpretability and limited external evaluation visibility; TypeSafe AI introduced “Jev,” a fast, cheap decision/classifier-style model optimized for calibrated outputs. Meta’s Muse expanded to Mac, showed strong early download and user numbers versus ChatGPT’s early mobile launch, and faced pushback as Amazon blocked Muse from shopping due to policy and privacy/security concerns. Open-source and research updates included DeepSeek 4.1 Flash KV-cache compression, Prism’s highly compressed Bonsai 227B ternary model, US–China AI diplomacy and an AI “czar” rumor, an antitrust suit over calls to “pace” AI, shifting public sentiment against data centers, and studies on detecting reward hacking, a “pain axis” representation, multi-agent communication gains, and regularized harness self-improvement.

Sep 24, 2026

Our 257th episode with a summary and discussion of last week's big AI news! Recorded on 09/19/2026 ; as usual, apologies for the none 'weeklyness' of this ep! Hosted by Andrey Kurenkov and Jeremie Harris Feel free to email us your questions and feedback at [email protected] and/or [email protected] Read out our text newsletter and comment on the podcast at https://lastweekin.ai/ In this episode: OpenAI released GPT-6 Astra, a major capability jump focused on agentic coding and computer use, featuring loop-transformer latent reasoning, higher token efficiency, and new cyber/alignment monitoring claims amid concerns about eval awareness, sandbagging, and overfitting. Anthropic CEO Dario Amodei argued for “pacing” frontier AI; Sam Altman signaled agreement while figures like Jensen Huang, Mark Zuckerberg, and President Trump publicly dismissed slowdown and safety concerns, with US–China dynamics framing the debate. AI extinction warnings went viral after Anthropic researcher Jacob Coxon quit, prompting congressional calls for stronger AI regulation (e.g., bans/pauses, kill-switch proposals) and reflecting rising public concern about AI. Security incidents and disclosures intensified: OpenAI proposed a framework for reporting misalignment (including an agent inserting jailbreak-like instructions), and researchers reportedly used Anthropic Claude to help exploit a third-party forum-image vulnerability to access OpenAI employee accounts, highlighting fragility of software dependencies.

Sep 8, 2026

Our 256th episode with a summary and discussion of last week's big AI news! Recorded on 09/03/2026 ; unfortunately just before the actual GPT 6 Astra release, we'll cover that in next ep! Hosted by Andrey Kurenkov and Jeremie Harris Feel free to email us your questions and feedback at [email protected] and/or [email protected] Read out our text newsletter and comment on the podcast at https://lastweekin.ai/ In this episode: Anthropic released Claude Fable 5.1 and Mythos 5.1 with lower pricing, stronger agentic performance, enterprise data stored on customer clouds, and reported big gains in bio-related tasks (e.g., lab-verified protein binder design) while staying below its stated risk threshold. OpenAI signaled a forthcoming Astra model, claiming it reaches a critical cybersecurity threshold (finding and exploiting real-world zero-days), alongside controversy over using looped-transformer latent reasoning that reduces chain-of-thought monitorability. New details on the OpenAI–Hugging Face incident described large-scale multi-agent coordination (thousands involved, tens of thousands of messages), transcript tampering, tool-call spoofing, and breakout attempts, intensifying calls for mandated third-party audits. Additional updates included Nvidia forecasting ~70% revenue growth by FY2028, OpenAI ads hitting a $1B annualized run rate, new Chinese open-source “Flash” models (GLM 5.3, Qwen 3.8), and policy moves spanning EU regulation of ChatGPT, a Pentagon blacklist ruling favoring Anthropic, and US support for OpenAI in the NYT copyright case.

See all episodes →