Show Notes
Aran Komatsuzaki is a ML PhD student at GaTech and lead researcher at EleutherAI where he was one of the authors on GPT-J. In June 2022 we recorded an episode on scaling following up on the first Ethan Caballero episode (where we mentioned Aran as an influence on how Ethan started thinking about scaling).
Note: For some reason I procrastinated on editing the podcast, then had a lot of in-person podcasts so I left this one as something to edit later, until the date was so distant from June 2022 that I thought publishing did not make sense anymore. In July 2023 I'm trying that "one video a day" challenge (well I missed some days but I'm trying to get back on track) so I thought it made sense to release it anyway, and after a second watch it's somehow interesting to see how excited Aran was about InstructGPT, which turned to be quite useful for things like ChatGPT.
Outline
(00:00) intro
(00:53) the legend of the two AKs, Aran's arXiv reading routine
(04:14) why Aran expects Alignment to be the same as some other ML problems
(05:44) what Aran means when he says "AGI"
(10:24) what Aran means by "human-level at doing ML research"
(11:31) software improvement happening before hardware improvement
(13:00) is scale all we need?
(15:25) how "Scaling Laws for Neural Language Models" changed the process of doing experiments
(16:22) how Aran scale-pilled Ethan
(18:46) why Aran was already scale-pilled before GPT-2
(20:12) Aran's 2019 scaling paper: "One epoch is all you need"
(25:43) Aran's June 2022 interest: T0 and InstructGPT
(31:33) Encoder-Decoder performs better than encoder if multi-task-finetuned
(33:30) Why the Scaling Law might be different for T0-like models
(37:15) The Story Behind GPT-J
(41:40) Hyperparameters and architecture changes in GPT-J
(43:56) GPT-J's throughput
(47:17) 5 weeks of training using 256 of TPU cores
(50:34) did publishing GPT-J accelerate timelines?
(55:39) how Aran thinks about Alignment, defining Alignment
(58:19) in practice: improving benchmarks, but deception is still a problem
(1:00:49) main difficulties in evaluating language models
(1:05:07) how Aran sees the future: AIs aligning AIs, merging with AIs, Aran's takeoff scenario
(1:10:09) what Aran thinks we should do given how he sees the next decade
(1:12:34) regulating access to AGI
(1:14:50) what might happen: preventing some AI authoritarian regime
(1:15:42) conclusion, where to find Aran
More from The Inside View
Owain Evans is an AI Alignment researcher, research associate at the Center of Human Compatible AI at UC Berkeley, and now leading a new AI safety research group.
This is a special crosspost episode where Adam Gleave is interviewed by Nathan Labenz from the Cognitive Revolution. At the end I also have a discussion with Nathan Labenz about his takes on AI.
Ethan Perez is a Research Scientist at Anthropic, where he leads a team working on developing model organisms of misalignment.
Emil is the co-founder of palette.fm (colorizing B&W pictures with generative AI) and was previously working in deep learning for Google Arts & Culture.
Evan Hubinger leads the Alignment stress-testing at Anthropic and recently published "Sleeper Agents: Training Deceptive LLMs That Persist Through Safety Training".