Show Notes
One popular objection to AI concerns is to declare that LLMs can never be AGI. You need a "new paradigm". Therefore, AGI is so far in the future that it's not worth worrying about. A common counterargument is to claim that no, LLMs can become AGI. But even without that counterargument, I think the "therefore" fails on its own terms. The key question is: how much of a new paradigm do we need? The landmark discoveries on the road to modern LLMs are something like: 1950s: Neural networks 1967: Multi-layer perceptron 2010: Modern deep learning 2017: Transformer, LLM 2022: RLHF, chatbots 2024: Chain of thought / test-time compute We can think of this as an "evolutionary tree", where a given LLM (let's say Claude Opus 4.7) shares a recent "common ancestor" with all other chatbots, and only a very distant "common ancestor" with everything else descended from the multi-layer perceptron. If AGI needs a "new paradigm", what common ancestor can we expect AGI and LLMs to share? AGI will very likely use neural networks, because the human brain is a neural network and qualifies as an AGI. It will probably use deep learning, because although deep learning isn't exactly analogous to the brain, it seems like a pretty reasonable way to emulate the brain's learning algorithms onto computer hardware. Skeptics like Yann LeCun and Gary Marcus usually pinpoint LLMs/transformers as the step where we went wrong. LeCun thinks that the first AGIs may be within the deep learning paradigm (but not LLMs); Marcus thinks that they'll combine insights from deep learning with something else. How soon should we expect a new paradigm as revolutionary as LLMs/transformers? Since we got LLMs/transformers nine years ago, Lindy's Law suggests nine more years. How soon should we expect a new paradigm as revolutionary as deep learning? By the same logic, sixteen years from now. Lindy's Law has a heavy tail, which means we can't simply halve these to find our 25th percentile estimate. Our 25th percentile estimate for the next advance as exciting as LLMs should be three years from now; for deep learning, it's five years. So even if you think AGI will require a further paradigm shift as big as the invention of the LLM or as deep learning itself, you should have 25% chance it will be developed in the next 3 - 5 years. Which is about as long as the LLM-only crowd think things will take! This isn't an excuse for relegating the risk of AGI to some vague indefinite future. It could still be the late 2020s or early 2030s! (Might we expect that low-hanging-fruit effects make the next paradigm harder to find than the last one? In practice, fields get more researchers as time goes on, and that effect usually causes time-between-advances to be approximately constant. And in fact, the number of AI researchers has grown at an unprecedented pace for a scientific field, and growth will enter an even faster regime once AIs themselves can contribute. Overall these make me think things will go even faster than Lindy's Law predicts - but I think Lindy's Law is a useful upper bound.) (Would there still be a long time between the invention of the new paradigm and the point where it could be used to maximum effect? It took five years between the invention of the transformer and ChatGPT, the first commercially-successful transformer-based project. But most of that time was spent scaling up, and we've already scaled up. If we invent a new paradigm in 2030, then any frontier lab willing to bet on it can quickly provide it with levels of compute sufficient to train human-brain-sized models.) This is my attempt to talk to the new-paradigm-wanters in their own language, but I think there's also a subtler point that undermines this worldview. In the past, new paradigms have proven useful in allowing scaling to continue after an old paradigm passed the regime where it could efficiently convert scale to results. LLMs still seem to be able to convert scale to results; while this conti
More from Astral Codex Ten Podcast
Finalist #8 in the Book Review Contest This is one of the finalists in the 2026 book review contest, written by an ACX reader who will remain anonymous until after voting is done. I'll be posting about one of these a week for several months. When you've read them all, I'll ask you to vote for a favorite, so remember which ones you liked] [Content warning: Sexual and relationship practices from a thousand years ago, from a time and place with a very different morality than what you probably subscribe to.] The past is an alien civilisation Around the year 1000, the Imperial Court of Japan in Kyoto was a world of opulence, aesthetics, and ambition. Isolated from the lives of regular commoners, every aspect of the lives of the elite were governed by complex etiquette rules, and relationships were thoroughly intertwined with politics and the ambition for power and wealth. High-ranking women and men lived lives so separate that they might have a passionate love affair without properly seeing each other's faces. It was in this world that Murasaki Shikibu, an intelligent and educated lady in Empress Shōshi's court, wrote The Tale of Genji, which is widely considered to be the world's oldest novel. It is an extraordinary masterpiece and a highly questionable soap opera, and I've never read anything like it. https://www.astralcodexten.com/p/your-book-review-the-tale-of-genji
I am Daniel Böttger1. In 2023, my review of On The Marble Cliffs took second in the ACX book review contest, and Scott offered me the right to pitch him more essays. I have worked as a neuroscientist, so in 2024 I pitched a post with my theory of consciousness. Later that year, my connection to neuroscience took an unexpected turn, and I wrote about my experience getting diagnosed with brain cancer. This post is an update on how I'm doing. I'm doing well. Very well, in fact. Suspiciously well. So well that the only other person I've ever heard of doing as well as I am was Jo Cameron, who was profiled on ACX for her rare genetic mutation that makes her incapable of negative emotion. My case can't be a mutation, so it must be related to my brain cancer and subsequent surgery. I write about it here in the hopes of making a novel contribution to the study of low-suffering phenotypes. Content warning: the first part of this review contains medical details that some people might find gruesome; I am bad at judging these things since I can no longer feel disgust or fear myself, and have tried to err on the side of completeness. https://www.astralcodexten.com/p/absurd-adventure-and-amygdalectomy
I. Econblogger Nicholas Decker is not a fan of AI 2040: One of the centerpieces of AI 2040 is the proposal that we pause the development of AI at a high level, use the AI to research alignment, and only then proceed. I am skeptical of this, and not only because it seems impractical to get such cooperation. My first argument is that I do not believe a period of "researching alignment" is meaningful – alignment research is simply capabilities research, patching particular bugs in the system, and we're not going to find anything deep without actually being in contact with the systems […] I envision AI alignment as being like research into aviation safety. There is no grand theory of why airplanes crash, unless you take that to mean gravity. Instead, we patch little problems. The airplane crashed because of metal fatigue in the engine – okay, we mandate inspections of engines, and prevent crashes from occurring by that source. As new problems arise, we iterate. We don't even need for there to be a crash to make improvements – we test proactively, we build in redundancy, and we monitor for deviations which could be a threat. What we are not doing is thinking about how we are going to keep flying saucers from crashing. If flying saucers are invented, then we will work to keep them from crashing, but we will use exactly the same methods. We will test them, see how they perform, and patch particular problems. AI will be the same way […] We should expect alignment to happen by default. My challenge for Nicholas is: would this work on you? https://www.astralcodexten.com/p/nicholas-decker-in-hell
Finalist #7 in the Book Review Contest [This is one of the finalists in the 2026 book review contest, written by an ACX reader who will remain anonymous until after voting is done. I'll be posting about one of these a week for several months. When you've read them all, I'll ask you to vote for a favorite, so remember which ones you liked] This is not a review of Rabelais. This is a review of a certain, incomplete translation of Rabelais. It is also arguably one of the greatest translations in our language. Let's meet the hero of our story. https://www.astralcodexten.com/p/your-book-review-thomas-urquharts
[Original post: Why I'm Staying Out Of The Substack Religion Debate] Ben Smith writes: This post sure reads like participating in the object level debate, not just "writing a post about how I won't participate". I did joke about this at the end, but I also want to defend calling this a post about how I won't participate. This essay presented a kind of hand-wavey heuristic for why I'm not spending lots of time examining the details of every individual argument. This is fine, everyone has to do this or be overwhelmed, but I think it's important to keep hand-wavey heuristics for not participating in arguments, vs. actual participation in arguments, distinct. Little annoys me more than when I spend weeks carefully gathering citations and making a substantive contribution in a domain, and somebody responds with "well that's obviously all bullshit because it vaguely reminds me of something I have negative affect towards". It's not necessarily wrong to deprioritize a topic because it vaguely reminds you of something that you have negative affect toward, but I would prefer these people admit they're dismissing/ignoring it rather than claim to be engaging with it. I'm working on a longer post about this topic. https://www.astralcodexten.com/p/highlights-from-the-comments-on-the-491