The Science of Learning to Code

I texted Don about a side gig: €5,000 for every 1% you shave off a zip file. It's a real contest. For 20 years, Marcus Hutter has offered €500,000 to anyone who can losslessly compress a gigabyte of Wikipedia below 110 megabytes. Gzip gets you to 322. After that you're on your own. So we tried. Run-length encoding. Pointing back instead of repeating yourself. Huffman coding worked out by hand. Arithmetic coding, which spends less than one bit per character. Then Claude Shannon's 1951 guessing game, and a paper where an LLM beats every record and still wins nothing, because the model counts against you. Don wanted the euros. I had other reasons for bringing him over. Episode Page Support The Show Subscribe To The Podcast Join The Newsletter

I've been trying to understand how machine learning actually works. Not use it, understand it, down to the ifs and loops. How does a program built out of plain conditionals get better on its own? So late one night I sent Don a paper. Three words in the title: reward is enough. The claim is that all of intelligence, the whole thing, comes down to a system maximizing a reward. Don thought that was far too reductive. I wanted to pull it apart and see if it held up.

I've been confusing Don with frontier-lab links late at night for a bit. Ilya Sutskever told a NeurIPS audience that pre-training as we know it would unquestionably end. There's only one internet, and the data isn't growing. The frontier labs call this the pre-training wall. A leaked Google memo from 2023 argued they had no moat. R1 is on GitHub. Llama is on Hugging Face. OpenAI's secondary-market valuation has climbed past $850 billion. Don was confused. So he came over and we made an episode about it. Episode Page Support The Show Subscribe To The Podcast Join The Newsletter

Kate Gregory has been writing C++ for over forty years. Books, keynotes, a consulting firm she built from the ground up. At sixty-three, she's one of the most experienced programmers alive. She surveyed hundreds of software engineers about getting older. What scares you? What's changed? What have you lost? The things people feared most — memory, stamina, keeping up — weren't the real threats. The stuff that was actually breaking down was mostly fixable. A bad knee wasn't aging, it was a torn cartilage. Wrist pain disappeared when she changed how she slept. But buried in the research was something harder to fix. The single factor that predicted whether you'd age well or badly had nothing to do with your body at all. The opponent isn't aging. The opponent is the story about aging. Episode Page Support The Show Subscribe To The Podcast Join The Newsletter

Corey told me about his AI cat reel problem. He found these AI-genearted cat videos hilarious. Who makes these? He kept sending them to his wife. Then he tried to stop watching and he couldn't. So I went down the rabbit hole of how social media algorithms actually work. It starts simple. Upvote, downvote, sort by time. But by 2017 Facebook has a metric that quietly reshapes what two billion people see. Then a leaked playbook lands, and a CEO takes the stand in Los Angeles. Today is an investigation into what happens when the algorithm knows you better than you know yourself. Episode Page Support The Show Subscribe To The Podcast Join The Newsletter