Show Notes
📌 Copy the AI system I use to run a one-person $1M+ business: https://www.behindthecraft.com 🤝 Subscribe to my practical AI newsletter read by 150K+ professionals: https://creatoreconomy.so
Shreya and Hamel teach the industry standard course for AI evaluations taken by 4,500+ students from OpenAI, Google, and more. In our interview, I asked them to give live feedback on the AI evals that I built for my skills and demo how anyone can run evals with ChatGPT/Claude and their free evals skill. Don’t miss this up to date primer on evals from the real experts. 🙂
P.S., I asked Shreya and Hamel to give my audience 25% off the next cohort of their AI evals course. It has 4.7 stars with 900 reviews which is almost unheard of on Maven. Sign up via the link here: https://maven.com/parlance-labs/evals?utm_campaign=peteryang&utm_medium=affiliate&utm_source=maven&promoCode=PETERYANG
We talked about: (00:00) Why AI evals are completely different now with the latest models (02:30) Demo: Live audit of the evals that I built for my AI skills (05:14) Top down vs. bottom up evals and why AI sucks at the latter (08:16) Spinning up AI agents to grade each eval criteria (14:32) Demo: Using Shreyas AI skill to run evals in Claude (23:46) How to review output labels to identify failures (34:37) How to turn failures into reusable evals (43:19) Where human judgement is still needed
Resources mentioned: Free eval skills from Shreya: https://github.com/ai-evals-course/evals-skills Do automated evals work?: https://parlance-labs.com/blog/posts/auto-evals/
Thanks to our sponsors: Wispr Flow — 4x faster than typing with your voice — https://ref.wisprflow.ai/peteryang
📌 Get the takeaways: https://creatoreconomy.so/p/how-to-build-better-ai-evals-with-claude-code-shreya-hamel
Where to find Shreya and Hamel: Shreya: https://www.sh-reya.com/ Hamel: https://hamel.dev/
Subscribe to this channel - more interviews coming soon!
More from Behind the Craft
Pomp and I worked together at Facebook a decade ago before he became an investor, best-selling author, and public company CEO with an audience of 3.3M. We had a great chat about how to build an extraordinary career and life and how he's building Silvia, an AI CFO for independent investors, to take on the AI labs.
Micky is a full-stack engineer who has taught 100K+ people how to build with AI. In our episode, he walked me through his 4-step agentic engineering workflow to avoid pumping out slop: Isolate every task, build to a clear structure, prove via validation loops, and ship after agents review your PRs or escalate to you for a manual look.
Sam is the co-founder of Granola, the AI meeting notes app now valued at $1.5B. Granola has helped me survive (and even enjoy) countless meetings, so I was excited to learn how Sam uses AI to show up prepared. We also talked about building an AI-native company without the 996 grind and what design means when agents skip your app entirely.
Copy my best AI workflows to save time and automate busywork: https://www.behindthecraft.com 🤝 Subscribe to my practical AI newsletter read by 150K+ professionals: https://creatoreconomy.so
Ethan is the product lead for ChatGPT Finances at OpenAI. He previously co-founded Digit, which helped more than 10 million people save over $9 billion. In our interview, he shared six practical ways to use ChatGPT to find wasted spending, save money on taxes, maximize credit card rewards, and model major financial decisions. I’m confident at least one of these use cases will save you money today.