Show Notes
Ethan is known on Twitter as the edgiest person at MILA. We discuss all the gossips around scaling large language models in what will be later known as the Edward Snowden moment of Deep Learning. On his free time, Ethan is a Master’s degree student at MILA in Montreal, and has published papers on out of distribution generalization and robustness generalization, accepted both as oral presentations and spotlight presentations at ICML and NeurIPS. Ethan has recently been thinking about scaling laws, both as an organizer and speaker for the 1st Neural Scaling Laws Workshop.
Transcript: https://theinsideview.github.io/ethan
Youtube: https://youtu.be/UPlv-lFWITI
Michaël: https://twitter.com/MichaelTrazzi
Ethan: https://twitter.com/ethancaballero
Outline
(00:00) highlights
(00:50) who is Ethan, scaling laws T-shirts
(02:30) scaling, upstream, downstream, alignment and AGI
(05:58) AI timelines, AlphaCode, Math scaling, PaLM
(07:56) Chinchilla scaling laws
(11:22) limits of scaling, Copilot, generative coding, code data
(15:50) Youtube scaling laws, constrative type thing
(20:55) AGI race, funding, supercomputers
(24:00) Scaling at Google
(25:10) gossips, private research, GPT-4
(27:40) why Ethan was did not update on PaLM, hardware bottleneck
(29:56) the fastest path, the best funding model for supercomputers
(31:14) EA, OpenAI, Anthropics, publishing research, GPT-4
(33:45) a zillion language model startups from ex-Googlers
(38:07) Ethan's journey in scaling, early days
(40:08) making progress on an academic budget, scaling laws research
(41:22) all alignment is inverse scaling problems
(45:16) predicting scaling laws, useful ai alignment research
(47:16) nitpicks aobut Ajeya Cotra's report, compute trends
(50:45) optimism, conclusion on alignment
More from The Inside View
Owain Evans is an AI Alignment researcher, research associate at the Center of Human Compatible AI at UC Berkeley, and now leading a new AI safety research group.
This is a special crosspost episode where Adam Gleave is interviewed by Nathan Labenz from the Cognitive Revolution. At the end I also have a discussion with Nathan Labenz about his takes on AI.
Ethan Perez is a Research Scientist at Anthropic, where he leads a team working on developing model organisms of misalignment.
Emil is the co-founder of palette.fm (colorizing B&W pictures with generative AI) and was previously working in deep learning for Google Arts & Culture.
Evan Hubinger leads the Alignment stress-testing at Anthropic and recently published "Sleeper Agents: Training Deceptive LLMs That Persist Through Safety Training".