Show Notes
We're back! In Episode 2, Anton Teaches Packy about Deepmind's March 2022 paper, Training Compute-Optimal Large Language Models, or as it's more commonly known, Chinchilla. Prior to Chinchilla, the best way to improve the performance of LLMs was thought to be by scaling up the size of the model. As a result, the largest models now have over 500 billion parameters. But there are only so many GPUs in the world, and throwing compute at the problem is expensive and energy intensive. In this paper, Deepmind found that the optimal way to scale an LLM is actually by scaling size (parameters) and training (data) proportionally. Given the race for size, today's models are plenty big but need a lot more data.
In this conversation, we go deep on the paper itself, but we also zoom out to talk about the politics of AI, when AGI is going to hit, where to get more data, and why AI won't take our jobs. This one gets a lot more philosophical than our first episode as we explore the implications of Chinchilla and LLMs more generally. If you enjoyed this conversation, subscribe for more. We're going to try to release one episode per week, and we want to make this the best way to get a deeper understanding of the mind-blowing progress happening in AI and what it means for everything we do as humans.
LINKS: Training Compute-Optimal Large Language Models: https://arxiv.org/abs/2203.15556
chinchilla's wild implications: https://www.lesswrong.com/posts/6Fpvc...
Scaling Laws for Neural Language Models (Kaplan et al): https://arxiv.org/abs/2001.08361
---
Send in a voice message: https://podcasters.spotify.com/pod/show/ageofmiracles/message
More from "Age of Miracles"
Here's our full-length interview with Jordan Bramble and our own Julia DeWahl, co-founders of Antares, where we dive deep into how Antares plans on manufacturing in mass micro-sized nuclear reactors to provide abundant energy for all. 🔥 Apply to join over 400 founders and Execs in the Turpentine Network: https://hmplogxqz0y.typeform.com/to/JCkphVqj
Here's our full-length interview with David Kirtley, founder of Helion Energy, where we dive deep into the need for fusion as a replacement for fossil fuels. 🔥 Apply to join over 400 founders and Execs in the Turpentine Network: https://hmplogxqz0y.typeform.com/to/JCkphVqj
Here's our full-length interview with Bret Kugelmass, founder of Last Energy, where we dive deep into nuclear manufacturing and the importance of manufacturing efficiency. 🔥 Apply to join over 400 founders and Execs in the Turpentine Network: https://hmplogxqz0y.typeform.com/to/JCkphVqj
We're kicking off the first of our bonus episodes with Tyler Bernstein, founder and CEO of Zeno Power, where he shares in-depth how they're creating radioisotope power systems for lunar applications. 🔥 Apply to join over 400 founders and Execs in the Turpentine Network: https://hmplogxqz0y.typeform.com/to/JCkphVqj
We can live in an Age of Miracles, but it’s going to take a lot of work. Energy abundance is the most important thing we can work towards as a species. More than simply surviving climate change, we have the opportunity to unlock a new level of human thriving if we can successfully navigate this transition. That’s why we chose to do the first season of Age of Miracles on the energy sources of the future. In this episode of Age of Miracles, the final episode of season one, Packy and Julia reflect on the importance of energy abundance, how nuclear energy helps us unlock it, and what the future might look like when we do. Thank you to all of our guests this season for making this show possible—we're so grateful.