Show Notes
Contest links: General $50k https://www.chinatalk.media/p/50k-chinatalk-submission-hiring-contest we're looking at submissions on a rolling basis!
AI evals: Due
Over the past few years, we’ve seen hints of policymakers and national leaders using AI models in their actual policy decision-making. The Prime Minister of Sweden said he uses it for second opinions on policy; the German Chancellor is testing it to draft legislation; even Trump said he had it re-write a speech at some point. It’s fair to assume that senior leadership across the world, and in Washington as well, have started using AI not just for tactical or operational tasks, but increasingly for broad strategic decision-making.
While that’s exciting — there’s the promise of uplift and smarter calls on some of the most consequential decisions leaders face in foreign policy and national security — we’re also flying blind. Enormous effort and energy goes into benchmarking and evaluation for things like coding. The experiments you can run to make models better at software development are much easier to execute and much lower-stakes than running a real experiment when you’re deciding whether to invade a country or sign a treaty.
That’s why we at ChinaTalk are trying to kickstart a field aimed at helping researchers and policymakers understand exactly what they’re working with when they ask these models to support some of the most consequential decisions they may make in their lifetimes. We’re launching an evals/essay project contest to explore this theme.
I’ve brought on two expert AI eval creators to discuss why the field is important, what interesting work has already been done on how models approach broad national-security and strategic questions, and how you — as an eval professional, semi-professional, or just a concerned person — can contribute new ways to poke and prod at these models and see what they can really do.
Joining us today: Florian Brand, research engineer at Prime Intellect, and John Chen, professor at the University of Arizona, who’s done some pretty wild things getting models to start nuclear wars with each other in Civ V.
Our conversation covers:
How frontier labs are hitting eval-building limits — they’ve gone from undergrads to PhDs to field experts, and the models are now catching the experts’ mistakes.
How Civilization V exposes AI’s strategic blind spots (terrible second-order reasoning) and models’ distinct strategic personalities (Claude’s really into science!).
Why ethical prompting in Civilization V still doesn’t stop models from launching nukes.
Jordan's PresidentBench eval, where a Chinese model was nonchalant about a Taiwan invasion, while Claude wanted to keep Taiwan free and independent.
Advice for designing better AI evals and ChinaTalk’s new essay/evals contest!
Learn more about your ad choices. Visit megaphone.fm/adchoices
More from ChinaTalk
Nick Brown is a co-founder of the Center for Technology and Statecraft (CTS), a new think tank spun out of IFP, that looks at long-term technology challenges and opportunities facing the US. Nick was a physics major, spent a while as a software engineer in and around government, and had an arc as an aspiring concert violinist. But we’re here to talk about what is in pole position for my paper of the year: his master class on what would happen if the US cut off DUV exports to China: DUV Immersion Lithography: The Foundation of US AI Compute Advantage. Co-hosting is ChinaTalk’s Aqib Zakaria.We discuss:
He was recently promoted to DIU’s acting director after spending the past year running the Drone Dominance Program. Co-hosting are Brian Clark, senior fellow at Hudson Institute and a retired submariner, and Justin McIntosh, a twenty-year retired Green Beret who served in the 1st and 5th Groups, who now works in defense tech and writes a Substack called Mind of Things.
Richard Bian is the head of product operations for Ant Ling, the foundational LLM at Ant. If you don’t know Ant, they most famously make Alipay, which makes your money and your world run when you’re in China and sometimes beyond. Richard is a former software engineer at Microsoft and Square, and he previously served as head of open source at Ant, where he scaled the open source initiative into a dedicated team. He’s lived and worked in Singapore, Canada, the US, and China, and he loves to build products.
Saif Khan makes his ChinaTalk debut. He is perhaps my favorite example of posting-to-policy that I’ve seen over the course of my time following Washington. Saif wrote some truly excellent deep dives into AI and semiconductors during his time at CSET, then went into the NSC and Commerce in the early-to-mid 2020s, across both the Biden and Trump II administrations, to help manifest the export control regulatory regime we have today. He is now out of government and launching the Center for Technology and Statecraft, an IFP-affiliated think tank, which is already producing excellent, grounded, technical, forward-looking work that will certainly shape many an AI policy discussion in the years to come.
I join Matt and Jerusalem for an unhinged conversation (most entertaining of the year?) about China, AI, corruption, politics, and various other things.