Show Notes
Summary In this episode of the AI Engineering Podcast Jamie De Guerre, founding SVP of product at Together.ai, explores the role of open models in the AI economy. As a veteran of the AI industry, including his time leading product marketing for AI and machine learning at Apple, Jamie shares insights on the challenges and opportunities of operating open models at speed and scale. He delves into the importance of open source in AI, the evolution of the open model ecosystem, and how Together.ai's AI acceleration cloud is contributing to this movement with a focus on performance and efficiency.
Announcements Hello and welcome to the AI Engineering Podcast, your guide to the fast-moving world of building scalable and maintainable AI systems Your host is Tobias Macey and today I'm interviewing Jamie de Guerre about the role of open models in the AI economy and how to operate them at speed and at scale Interview Introduction How did you get involved in machine learning? Can you describe what Together AI is and the story behind it?What are the key goals of the company?
The initial rounds of open models were largely driven by massive tech companies. How would you characterize the current state of the ecosystem that is driving the creation and evolution of open models? There was also a lot of argument about what "open source" and "open" means in the context of ML/AI models, and the different variations of licenses being attached to them (e.g. the Meta license for Llama models). What is the current state of the language used and understanding of the restrictions/freedoms afforded? What are the phases of organizational/technical evolution from initial use of open models through fine-tuning, to custom model development? Can you outline the technical challenges companies face when trying to train or run inference on large open models themselves?What factors should a company consider when deciding whether to fine-tune an existing open model versus attempting to train a specialized one from scratch?
While Transformers dominate the LLM landscape, there's ongoing research into alternative architectures. Are you seeing significant interest or adoption of non-Transformer architectures for specific use cases? When might those other architectures be a better choice?
While open models offer tremendous advantages like transparency, control, and cost-effectiveness, are there scenarios where relying solely on them might be disadvantageous?When might proprietary models or a hybrid approach still be the better choice for a specific problem?
Building and scaling AI infrastructure is notoriously complex. What are the most significant technical or strategic challenges you've encountered at Together AI while enabling scalable access to open models for your users? What are the most interesting, innovative, or unexpected ways that you have seen open models/the TogetherAI platform used? What are the most interesting, unexpected, or challenging lessons that you have learned while working on powering AI model training and inference? Where do you see the open model space heading in the next 1-2 years? Any specific trends or breakthroughs you anticipate? Contact Info LinkedIn Parting Question From your perspective, what are the biggest gaps in tooling, technology, or training for AI systems today? Closing Announcements Thank you for listening! Don't forget to check out our other shows. The Data Engineering Podcast covers the latest on modern data management. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes. If you've learned something or tried out a project from the show then tell us about it! Email [email protected] with your story. To help other people find the show please leave a review on iTunes and tell your friends and co-workers. Links Together AI Fine Tuning Post-Training Salesforce Research Mi
More from AI Engineering Podcast
Summary In this episode, I spoke with Jonathan Ellis about how AI agents are changing the economics of software development and forcing teams to rethink where human time is best spent. We explored his argument that, now that code generation is dramatically faster, traditional human-centered code review is becoming a poor use of attention. Instead, Jonathan makes the case for shifting review earlier in the lifecycle toward specifications, RFCs, and architecture decisions, while investing more heavily in automated validation, static analysis, fuzzing, and other quality gates that can help verify whether software does what it is supposed to do regardless of how the code was written. Jonathan also shared lessons from building Brokk AI’s tooling, including Bifrost for fast, agent-friendly static analysis and Mjolnir for coordinating multiple coding agents across harnesses and subscriptions. We discussed the emerging need for better agent-oriented collaboration systems, the risks of blindly outsourcing judgment to AI, and the challenge of training newer engineers in a world where agents can do much of the implementation work. Overall, this conversation examines what remains uniquely valuable for humans as agents take over more of the mechanics of software delivery: setting goals, making product and design decisions, building better feedback loops, and deciding what is worth building in the first place.
Summary In this episode Ofer Mendelevitch shares what it really takes to evaluate RAG systems in production when your source data is incomplete, constantly changing, or difficult to validate against a clean ground truth. He explores how RAG has evolved from simple “chat with your PDF” demos into enterprise-grade retrieval systems that require robust ingestion pipelines, hybrid search, reranking, multimodal support, access controls, and refresh strategies for large and dynamic document collections. Ofer also explains why evaluation becomes one of the hardest parts of the stack, particularly when retrieval quality, generation quality, and system behavior can all shift as you change chunking, models, prompts, or ranking components. He digs into practical approaches for offline and online evaluation, including synthetic question generation, sampling production traffic, and LLM-as-a-judge techniques such as UMBRELA and Auto Nuggetizer. Along the way, he discusses the rise of agentic RAG, context engineering, MCP-connected tools, text-to-SQL systems, and the growing role of coding agents in building and maintaining AI pipelines. Ofer closes with his perspective on where retrieval, memory, multimodal systems, and agent orchestration are headed next, along with a reminder that the biggest gaps in AI today are often not just technical, but educational and organizational as well. Announcements Hello and welcome to the AI Engineering Podcast, your guide to the fast-moving world of building scalable and maintainable AI systems Your host is Tobias Macey and today I'm interviewing Ofer Mendelevitch about how to evaluate your RAG systems when you have incomplete or changing data to grade against Interview Introduction How did you get involved in machine learning? RAG was one of the first concepts to gain traction in the early explosion of LLMs and their application to real-world problems. How would you characterize its role in the current landscape? RAG is conceptually straight-forward, but can be quite complex to do well. What are some of the common pitfalls that teams encounter when bringing a retrieval-oriented LLM service into production? What are some of the predominant methods to avoid regressions or increased error rates as the data, code, and models change in a given deployment?There are numerous software projects and companies that offer various forms of evaluation/validation for RAG workloads. What are the factors that teams should be evaluating when selecting which one(s) to adopt?
Summary In this episode Nikunj Bajaj, co-founder and CEO of TrueFoundry, talks about the challenge of building reliable agents on top of inherently variable foundation models. He explores the idea of the agent harness as everything around the model engine: memory and context management, tool and MCP integration, sandboxed code execution, permissions, observability, and guardrails. Nikunj explained how TrueFoundry approaches enterprise AI as a centralized control plane for token traffic, while TrueForge provides an open source, vendor-neutral harness for building agents without locking teams into a proprietary stack. He talks about the complexity of model routing across both the harness and gateway layers, the tradeoffs between unified APIs and provider-specific capabilities, and the architectural decisions that affect cost, accuracy, and reliability. He highlighted the importance of benchmark-driven validation, durable execution, human-in-the-loop workflows, sandbox security, and especially agent identity and least-privilege access across chains of agents. Overall, this was a deep look at harness engineering as an emerging discipline, and why organizations need more than isolated point solutions if they want to operate agents safely, efficiently, and at scale. Announcements Hello and welcome to the AI Engineering Podcast, your guide to the fast-moving world of building scalable and maintainable AI systems Your host is Tobias Macey and today I'm interviewing Nikunj Bajaj about the challenges of keeping your agents reliable when the underlying models aren't and how your harness can help
Summary In this episode of the AI Engineering Podcast, Steven Watt, leader of the Office of the CTO at Red Hat, discusses practical paths to achieving AI sovereignty for organizations. He shares his two-decade experience in AI, highlighting how governments are building GPU platforms and protected data hubs to maintain control over AI workloads. Steve emphasizes why self-managed infrastructure is becoming a strategic necessity as companies outgrow cloud costs and require tighter control over models, data, and compliance. The conversation explores the operational substrate for AI sovereignty, including Kubernetes as the scale-out backbone for LLM serving, bridging the gap with PyTorch ecosystems, observability and policy for non-deterministic systems, and emerging security needs such as confidential inference and agentic identity. They also discuss model and hardware optionality (GPUs, CPUs, and new accelerators), the growing demand for energy-efficient inference, and the importance of open models and post-training to create durable differentiation. Steve identifies access to GPUs as the biggest gap hindering sovereign AI adoption today, emphasizing the need for broad access to GPUs for AI workloads to thrive. The conversation also touches on evolving architectures beyond transformers, the interplay between AI and data sovereignty, consolidation pressures from pilot chaos to standardized platforms, and the societal triad of universities, startups, and sovereign infrastructure.
Summary In this episode of the AI Engineering Podcast, Aman Agarwal, creator of OpenLit, discusses the operational foundations required to run LLM-powered applications in production. He highlights common early blind spots teams face, including opaque model behavior, runaway token costs, and brittle prompt management, emphasizing that strong observability and cost tracking must be established before an MVP ships. Aman explains how OpenLit leverages OpenTelemetry for vendor-neutral tracing across models, tools, and data stores, and introduces features such as prompt and secret management with versioning, evaluation workflows (including LLM-as-a-judge), and fleet management for OpenTelemetry collectors. The conversation covers experimentation patterns, strategies to avoid vendor lock-in, and how detailed stepwise traces reshape system design and debugging. Aman also shares recent advancements like a Kubernetes operator for zero-code instrumentation, multi-database configurations for environment isolation, and integrations with platforms such as Grafana and Dash0. They conclude by discussing lessons learned from building in the open, prioritizing reliability, developer experience, and data security, and preview future work on context management and closing the loop from experimentation to prompt/dataset improvements.