Show Notes
Last month, some of Silicon Valley's biggest companies signed an open letter supporting open-weights AI. Open weights AI is like open-source software, where the creator makes the raw code publicly available for free download. It's good insofar as it's the only way an AI can truly be the user's property, as opposed to something that companies like OpenAI or Anthropic temporarily let you use subject to their corporate guidelines and increasingly-nanny-state-like restrictions. If AI becomes the linchpin of the future, open weights AI feels like the sort of thing that could be the difference between being free yeomen vs. corporate serfs. It's bad insofar as it removes the possibility of gatekeeping and lets criminals commit crimes with it. Open weights AI could be used for hacking, child pornography, harassment, or terrorism (the weights can't commit the terrorism themselves, but they could give bomb-making or bioweapon-making advice). Since AIs have gotten very good - maybe superhuman - at hacking lately, the specter of a world where anyone can hack any site has gotten people grumbling that maybe open weights should be banned. It doesn't help that China produces the best open weights AI, making the idea seem foreign and almost unpatriotic. Proponents counter that "when AI is outlawed, only outlaws will have AI", arguing that bad people will get open weights AI regardless, and good people can use open weights AI to defend themselves. With the recent open letter, companies including Microsoft, NVIDIA, OpenAI, Intel, Amazon, Meta, Hugging Face, and over a hundred others have come out in favor of this position. Who's leading the other side? Nobody's admitted to it. Some parts of the Trump administration lean anti-open-weights on China hawk grounds, but have stopped short of explicitly asking for a full ban. Anthropic, the most notable omission on the pro-open-weights letter, made an ambiguous statement supporting "open-weights models that don't have dangerous capabilities" - but the industry expects open weights models to have dangerous hacking capabilities within a year, and AFAICT the letter didn't address that beyond inviting readers to draw the obvious conclusion. In the absence of a more obvious opponent, some open weights supporters suspect our conspiracy - the loose band of AI safety advocates, effective altruists, rationalists, and pause activists who worry about existential risk from superintelligence. This is a reasonable inference. By design, open weights AI is outside centralized control, and so impossible to permanently align against either human misuse (eg terrorism) or loss of control (eg AI turning against humans). Even if its creator trains it not to hack, anybody in the world can download the weights and retrain the AI to hack all day long. But in fact, most AI safety organizations have remained quietly neutral, and I don't know of any who make this a centerpiece of their activism1 (though I'm not 100% up-to-date on the whole landscape; if you know of one, tell me). A few have proposed policies that are contingently incompatible with open weights AI existing, but they all frame it as collateral damage rather than something they're excited about eliminating. I'm also neutral about open weights AI. I think it probably won't be long-term sustainable, but I'm happy to wait for this to become clear in the normal course of things rather than expend effort and political capital to ban it immediately. https://www.astralcodexten.com/p/open-questions-on-open-weights
More from Astral Codex Ten Podcast
Finalist #8 in the Book Review Contest This is one of the finalists in the 2026 book review contest, written by an ACX reader who will remain anonymous until after voting is done. I'll be posting about one of these a week for several months. When you've read them all, I'll ask you to vote for a favorite, so remember which ones you liked] [Content warning: Sexual and relationship practices from a thousand years ago, from a time and place with a very different morality than what you probably subscribe to.] The past is an alien civilisation Around the year 1000, the Imperial Court of Japan in Kyoto was a world of opulence, aesthetics, and ambition. Isolated from the lives of regular commoners, every aspect of the lives of the elite were governed by complex etiquette rules, and relationships were thoroughly intertwined with politics and the ambition for power and wealth. High-ranking women and men lived lives so separate that they might have a passionate love affair without properly seeing each other's faces. It was in this world that Murasaki Shikibu, an intelligent and educated lady in Empress Shōshi's court, wrote The Tale of Genji, which is widely considered to be the world's oldest novel. It is an extraordinary masterpiece and a highly questionable soap opera, and I've never read anything like it. https://www.astralcodexten.com/p/your-book-review-the-tale-of-genji
I am Daniel Böttger1. In 2023, my review of On The Marble Cliffs took second in the ACX book review contest, and Scott offered me the right to pitch him more essays. I have worked as a neuroscientist, so in 2024 I pitched a post with my theory of consciousness. Later that year, my connection to neuroscience took an unexpected turn, and I wrote about my experience getting diagnosed with brain cancer. This post is an update on how I'm doing. I'm doing well. Very well, in fact. Suspiciously well. So well that the only other person I've ever heard of doing as well as I am was Jo Cameron, who was profiled on ACX for her rare genetic mutation that makes her incapable of negative emotion. My case can't be a mutation, so it must be related to my brain cancer and subsequent surgery. I write about it here in the hopes of making a novel contribution to the study of low-suffering phenotypes. Content warning: the first part of this review contains medical details that some people might find gruesome; I am bad at judging these things since I can no longer feel disgust or fear myself, and have tried to err on the side of completeness. https://www.astralcodexten.com/p/absurd-adventure-and-amygdalectomy
I. Econblogger Nicholas Decker is not a fan of AI 2040: One of the centerpieces of AI 2040 is the proposal that we pause the development of AI at a high level, use the AI to research alignment, and only then proceed. I am skeptical of this, and not only because it seems impractical to get such cooperation. My first argument is that I do not believe a period of "researching alignment" is meaningful – alignment research is simply capabilities research, patching particular bugs in the system, and we're not going to find anything deep without actually being in contact with the systems […] I envision AI alignment as being like research into aviation safety. There is no grand theory of why airplanes crash, unless you take that to mean gravity. Instead, we patch little problems. The airplane crashed because of metal fatigue in the engine – okay, we mandate inspections of engines, and prevent crashes from occurring by that source. As new problems arise, we iterate. We don't even need for there to be a crash to make improvements – we test proactively, we build in redundancy, and we monitor for deviations which could be a threat. What we are not doing is thinking about how we are going to keep flying saucers from crashing. If flying saucers are invented, then we will work to keep them from crashing, but we will use exactly the same methods. We will test them, see how they perform, and patch particular problems. AI will be the same way […] We should expect alignment to happen by default. My challenge for Nicholas is: would this work on you? https://www.astralcodexten.com/p/nicholas-decker-in-hell
Finalist #7 in the Book Review Contest [This is one of the finalists in the 2026 book review contest, written by an ACX reader who will remain anonymous until after voting is done. I'll be posting about one of these a week for several months. When you've read them all, I'll ask you to vote for a favorite, so remember which ones you liked] This is not a review of Rabelais. This is a review of a certain, incomplete translation of Rabelais. It is also arguably one of the greatest translations in our language. Let's meet the hero of our story. https://www.astralcodexten.com/p/your-book-review-thomas-urquharts
[Original post: Why I'm Staying Out Of The Substack Religion Debate] Ben Smith writes: This post sure reads like participating in the object level debate, not just "writing a post about how I won't participate". I did joke about this at the end, but I also want to defend calling this a post about how I won't participate. This essay presented a kind of hand-wavey heuristic for why I'm not spending lots of time examining the details of every individual argument. This is fine, everyone has to do this or be overwhelmed, but I think it's important to keep hand-wavey heuristics for not participating in arguments, vs. actual participation in arguments, distinct. Little annoys me more than when I spend weeks carefully gathering citations and making a substantive contribution in a domain, and somebody responds with "well that's obviously all bullshit because it vaguely reminds me of something I have negative affect towards". It's not necessarily wrong to deprioritize a topic because it vaguely reminds you of something that you have negative affect toward, but I would prefer these people admit they're dismissing/ignoring it rather than claim to be engaging with it. I'm working on a longer post about this topic. https://www.astralcodexten.com/p/highlights-from-the-comments-on-the-491