Show Notes
Today on How AI Is Built, Nicolay Gerold sits down with Jorge Arango, an expert in information architecture. Jorge emphasizes that aligning systems with users' mental models is more important than optimizing backend logic alone. He shares a clear framework with four practical steps: Key Points: Information architecture should bridge user mental models with system data models Information's purpose is to help people make better choices and act more skillfully Well-designed systems create learnable (not just "intuitive") interfaces Context and domain boundaries significantly impact user understanding Progressive disclosure helps accommodate users with varying expertise levels Chapters 00:00 Introduction to Backend Systems 00:36 Guest Introduction: Jorge Arango 01:12 Podcast Dynamics and Guest Experiences 01:53 Timeless Principles in Technology 02:08 Interesting Conversations and Learnings 04:04 Physical vs. Digital Organization 04:21 Smart Defaults and System Maintenance 07:20 Data Models and Conceptual Structures 08:53 Designing User-Centric Systems 10:20 Challenges in Information Systems 10:35 Understanding Information and Choices 15:49 Clarity and Context in Design 26:36 Progressive Disclosure and User Research 37:05 The Role of Large Language Models 54:59 Future Directions and New Series (MLOps) Information Architecture Fundamentals What Is Information? Information helps people make better choices to act more skillfully Example: "No dog pooping" signs help predict consequences of actions Poor information systems fail to provide relevant guidance for users' needs Mental Models vs. Data Models Systems have underlying conceptual structures that should reflect user mental models Data models make these conceptual models "normative" in the infrastructure Designers serve as translators between user needs and technical implementation Goal: Users should think "the person who designed this really gets me" Design Strategies for Complex Systems Progressive Disclosure Present simple interfaces by default with clear paths to advanced functionality Example: HyperCard - visual interface for beginners with programming layer for experts Allows both novice and expert users to use the same system effectively Context Setting and Domain Boundaries All interactions happen within a context that influences understanding Words acquire different meanings in different contexts (e.g., "save" in computing vs. banking) Clearer domain boundaries make information architecture design easier Hardest systems to design: those serving many purposes for diverse audiences Conceptual Modeling (Underrated Practice) Should precede UI sketching but often skipped by designers Defines concepts needed in the system and their relationships Creates more cohesive and coherent systems, especially for complex projects More valuable than sitemaps, which imply rigid hierarchies LLMs and Information Architecture Current and Future Applications Transforming search experiences (e.g., Perplexity providing answers vs. link lists) Improving intent parsing in traditional search Helping information architects with content analysis and navigation structure design Enabling faster, better analysis of large content repositories Implementation Advice For Engineers and Designers Designers should understand how systems are built (materials of construction) Engineers benefit from understanding user perspectives and mental models Both disciplines have much to teach each other For Complex Applications Map conceptual models before writing code Test naming with real users Implement progressive disclosure with good defaults Remember: "If the user can't find it, it doesn't exist" Notable Quotes: "People only understand things relative to things they already understand." - Richard Saul Wurman "The hardest systems to design are the ones that are meant to do a lot of things for a lot of different people." - Jorge Arango "Very few things are intuitive. There's a long running joke in the industry that the
More from How AI Is Built
Nicolay here, Most AI coding conversations focus on which model to use. This one focuses on workflow - the specific commands, git strategies, and review processes that let one engineer ship production code with AI agents doing 80% of the work. Today I have the chance to talk to Kieran Klaassen, who built Cora (an AI email management tool) almost entirely solo using AI agents. His approach: treat AI agents like junior developers you manage, not tools you operate. The key insight centers on "compound engineering" - extracting reusable systems from every code review and interaction. Instead of just reviewing pull requests, Kieran records his review sessions with his colleague, transcribes them, and feeds the transcriptions to Claude to extract coding patterns and philosophical approaches into custom slash commands. In the podcast, we also touch on: Git worktrees for running multiple AI agents simultaneously The evolution from Cursor Composer to Claude Code and Friday Why pull request review is the real bottleneck, not code generation How to structure research phases to avoid AI going off the rails and more 💡 Core Concepts Compound Engineering: Extracting reusable systems, SOPs, and taste from every AI interaction - treating each code review or feature build as an opportunity to teach the AI your standards and decision-making patterns. Git Worktrees for AI Agents: Running multiple AI coding agents simultaneously by checking out different branches in separate file system directories, allowing parallel feature development without conflicts. Research-First AI Development: Starting every feature with a dedicated research phase where AI gathers context, explores multiple approaches, and creates detailed GitHub issues before any code is written. Tiered Code Review Systems: Implementing different review checklists and standards based on risk level (payments, migrations, etc.) with AI assistants handling initial passes before human review. 📶 Connect with Kieran: X / Twitter Cora 📶 Connect with Nicolay: Newsletter LinkedIn X / Twitter Bluesky Website My Agency Aisbach (for ai implementations / strategy) ⏱️ Important Moments The Sonnet 3.5 Breakthrough Moment: [09:30] Kieran describes vibe-coding a Swift app in one evening, realizing AI could support solo entrepreneurship for the first time. Building Cora's First Prototype: [12:45] One night to build a prototype that drafts email responses - the moment they knew there was something special about AI handling email. The Nice, France Experiment: [13:40] Testing automatic email archiving while walking around town, discovering the "calm feeling" that became Cora's core value proposition. Git Worktrees Discovery: [50:50] How Kieran discovered worktrees by asking AI for a solution to run multiple agents simultaneously, leading to his current parallel development workflow. Cursor 3.7 Breaking Point: [19:57] The moment Cursor became unusable after shipping too many changes at once, forcing the search for better agentic tools. Friday vs Claude Code Comparison: [22:23] Why Friday's "YOLO mode" and end-to-end pull request creation felt more like having a colleague than using a tool. Compound Engineering Philosophy: [33:18] Recording code review sessions and extracting engineering taste into reusable Claude commands for future development. The Research Phase Strategy: [04:48] Why starting with comprehensive GitHub issue research prevents AI agents from going off-rails during implementation. Pull Request Review Bottleneck: [28:44] How reviewing AI-generated code, not writing it, becomes the main constraint when scaling with agents. Multiple Agent Management: [48:14] Running Claude Code work trees in parallel terminals, treating each agent as a separate team member with distinct tasks. 🛠️ Tools & Tech Mentioned Claude Code Cursor Friday AI Git Worktrees Warp Terminal GitHub CLI Charlie AI (PR review bot) Context7 MCP Anthropic Prompt Improver
Nicolay here, while everyone races to cloud-scale LLMs, Pete Warden is solving AI problems by going completely offline. No network connectivity required. Today I have the chance to talk to Pete Warden, CEO of Useful Sensors and author of the TinyML book. His philosophy: if you can't explain to users exactly what happens to their data, your privacy model is broken. Key Insight: The Real World Action Gap LLMs excel at text-to-text transformations but fail catastrophically at connecting language to physical actions. There's nothing in the web corpus that teaches a model how "turn on the light" maps to sending a pin high on a microcontroller. This explains why every AI agent demo focuses on booking flights and API calls - those actions are documented in text. The moment you step off the web into real-world device control, even simple commands become impossible without custom training on action-to-outcome data. Pete's company builds speech-to-intent systems that skip text entirely, going directly from audio to device actions using embeddings trained on limited action sets. 💡 Core Concepts Speech-to-Intent: Direct audio-to-action mapping that bypasses text conversion, preserving ambiguity until final classification ML Sensors: Self-contained circuit boards processing sensitive data locally, outputting only simple signals without exposing raw video/audio Embedding-Based Action Matching: Vector representations mapping natural language variations to canonical device actions within constrained domains ⏱ Important Moments Real World Action Problem: [06:27] LLMs discuss turning on lights but lack training data connecting text commands to device control Apple Intelligence Challenges: [04:07] Design-led culture clashes with AI accuracy limitations Speech-to-Intent vs Speech-to-Text: [12:01] Breaking audio into text loses critical ambiguity information Limited Action Set Strategy: [15:30] Smart speakers succeed by constraining to ~3 functions rather than infinite commands 8-Bit Quantization: [33:12] Remains deployment sweet spot - processor instruction support matters more than compression On-Device Privacy: [47:00] Complete local processing provides explainable guarantees vs confusing hybrid systems 🛠 Tools & Tech Whisper: github.com/openai/whisper Moonshine: github.com/usefulsensors/moonshine TinyML Book: oreilly.com/library/view/tinyml/9781492052036 Stanford Edge ML: github.com/petewarden/stanford-edge-ml 📚 Resources Looking to Listen Paper: looking-to-listen.github.io Lottery Ticket Hypothesis: arxiv.org/abs/1803.03635 Connect: [email protected] | petewarden.com | usefulsensors.com Beta Opportunity: Moonshine browser implementation for client-side speech processing in JavaScript
Nicolay here,most AI conversations focus on training bigger models with more compute. This one explores the counterintuitive world where averaging weights from different models creates better performance than expensive post-training. Today I have the chance to talk to Maxime Labonne, who's a researcher at Liquid AI and the architect of some of the most popular open source models on Hugging Face. He went from researching neural networks for cybersecurity to building "Frankenstein models" through techniques that shouldn't work but consistently do. Key Insight: Model Merging as a Free LunchThe core breakthrough is deceptively simple: take two fine-tuned models, average their weights layer by layer, and often get better performance than either individual model. Maxime initially started writing an article to explain why this couldn't work, but his own experiments convinced him otherwise. The magic lies in knowledge compression and regularization. When you train a model multiple times on similar data, each run creates slightly different weight configurations due to training noise. Averaging these weights creates a smoother optimization path that avoids local minima. You can literally run model merging on a CPU - no GPUs required. In the podcast, we also touch on: Obliteration: removing safety refusal mechanisms without retraining Why synthetic data now comprises 90%+ of fine-tuning datasets The evaluation crisis and automated benchmarks missing real-world performance Chain of thought compression techniques for reasoning models 💡 Core Concepts Model Merging: Averaging weights across layers from multiple fine-tuned models to create improved performance without additional training Obliteration: Training-free method to remove refusal directions from models by computing activation differences Linear Merging: The least opinionated merging technique that simply averages weights with optional scaling factors Refusal Direction: The activation pattern that indicates when a model will output a safety refusal 📶 Connect with Maxime: X / Twitter: https://x.com/maximelabonne LinkedIn: https://www.linkedin.com/in/maxime-labonne/ Company: https://www.liquid.ai/ 📶 Connect with Nicolay: LinkedIn: https://www.linkedin.com/in/nicolay-gerold/ X / Twitter: https://x.com/nicolaygerold Website: https://www.nicolaygerold.com/ ⏱ Important Moments Model Merging Discovery Process: [00:00:30] Maxime explains how he started writing an article to debunk model merging Two Main Merging Use Cases: [11:04] Clear distinction between merging checkpoints versus combining different task-specific capabilities Linear Merging as Best Practice: [21:00] Why simple weight averaging consistently outperforms more complex techniques Layer Importance Hierarchy: [21:18] First and last layers have the most influence on model behavior Obliteration Technique Explained: [36:07] How to compute and subtract refusal directions from model activations Synthetic Data Dominance: [50:00] Modern fine-tuning uses 90%+ synthetic data 🛠 Tools & Tech Mentioned MergeKit: https://github.com/cg123/mergekit Transformer Lens: https://github.com/TransformerLensOrg/TransformerLens Hugging Face Transformers: https://github.com/huggingface/transformers PyTorch: https://pytorch.org/ 📚 Recommended Resources Maxime's Model Merging Articles: https://huggingface.co/blog/merge Model Soups Paper: https://arxiv.org/abs/2203.05482 Will Brown's Rubric Engineering: https://x.com/willccbb/status/1883611121577517092
Nicolay here, Most AI coding tools obsess over automating everything. This conversation focuses on the right balance between human skill and AI assistance - where manual context beats web search every time. Today I have the chance to talk to Ben Holmes, a software engineer at Warp, where they're building the AI-first terminal. Manual context engineering trumps automated web search for getting accurate results from coding assistants. Key Insight Expansion The breakthrough insight is brutally practical: manual context construction consistently outperforms automated web search when working with AI coding assistants. Instead of letting your AI tool search for documentation, find the right pages yourself and feed them directly into the model's context window. Ben demonstrated this with OpenAI's Realtime API documentation - after an hour of back-and-forth with web search, he manually found the correct API signatures and saved them as a reference file. When building new features, he attached this curated documentation directly, resulting in immediate success rather than repeated failures from outdated or incorrect search results. This approach works because you can verify documentation accuracy before feeding it to the AI, while web search often returns the first result regardless of quality or recency. In the podcast, we also touch on: Why React Native might become irrelevant as AI translation between native languages improves Model-specific strengths: Gemini excels at debugging while Claude dominates f unction calling The skill of working without AI assistance - "raw dogging" code for deep learning Warp's architecture using different models for planning (O1/O3) vs. coding (Claude/Gemini) 💡 Core Concepts Manual Context Engineering: Curating documentation, diagrams, and reference materials directly rather than relying on automated web search. Model-Specific Workflows: Matching AI models to their strengths - O1 for planning, Claude for f unction calling, Gemini for debugging. Raw Dog Programming: Coding without AI assistance to build f undamental skills in codebase navigation and problem-solving. Agent Mode Architecture: Multi-model system where Claude orchestrates task distribution to specialized agents through f unction calls. 📶 Connect with Ben: Twitter/X, YouTube, Discord (Warp Community), Website 📶 Connect with Nicolay: LinkedIn, X/Twitter, Bluesky, Website, [email protected] ⏱ Important Moments React Native's Potential O bsolescence: [08:42] AI translation between native languages could eliminate cross-platform frameworks Manual vs Automated Context: [51:42] Why manually curating documentation beats AI web search Raw Dog Programming Benefits: [12:00] Value of coding without AI assistance during Ben's first week at Warp Model-Specific Strengths: [26:00] Gemini's superior debugging vs Claude's speculative code fixes OpenAI Desktop App Advantage: [13:44] O utperforms Cursor for reading long files Warp's Multi-Model Architecture: [31:00] How Warp uses O1/O3 for planning, Claude for orchestration Function Calling Accuracy: [28:30] Claude outperforms other models at chaining f unction calls AI as Improv Partner: [56:06] Current AI says "yes and" to everything rather than pushing back 🛠 Tools & Tech Mentioned Warp Terminal, OpenAI Desktop App, Cursor, Cline, Go by Example, OpenAI Realtime API, MCP 📚 Recommended Resources Warp Discord Community, Ben's YouTube Channel, Go Programming Documentation 🔮 What's Next Next week, we continue exploring production AI implementations with more insights into getting generative AI systems deployed effectively. 💬 Join The Conversation Follow How AI Is Built on YouTube, Bluesky, or Spotify. Discord coming soon! ♻ Building the platform for engineers to share production experience. Pay it forward by sharing with one engineer facing similar challenges. ♻