Show Notes
In this insightful episode, hosts Justin Macorin and Bradley Arsenault chat with AI expert Abi Aryan to dive deep into the challenges of measuring the business impact of large language model applications. Abby draws from her 8 years of experience deploying and productizing LLMs to provide a comprehensive framework for defining relevant metrics.
She emphasizes the importance of bridging the gap between engineering and business objectives, highlighting that 76% of machine learning projects fail due to this disconnect. Abby shares practical advice on evaluating LLM performance through a combination of language skills assessment, task-specific metrics, human evaluation, and aligning with overarching business goals.
The conversation covers key topics such as measuring the efficacy of retrieval versus generation components, incorporating user feedback beyond simple thumbs-up/thumbs-downs, detecting and mitigating hallucinations, and tying metrics to concrete business KPIs like sales funnels and revenue generation.
--- Please check out Abi on LinkedIn at https://www.linkedin.com/in/goabiaryan/ Visit her website here: https://abiaryan.com/ Continue listening to The Prompt Desk Podcast for everything LLM & GPT, Prompt Engineering, Generative AI, and LLM Security. Check out PromptDesk.ai for an open-source prompt management tool. Check out Brad’s AI Consultancy at bradleyarsenault.me Add Justin Macorin and Bradley Arsenault on LinkedIn.
Please fill out our listener survey here to help us create a better podcast: https://docs.google.com/forms/d/e/1FAIpQLSfNjWlWyg8zROYmGX745a56AtagX_7cS16jyhjV2u_ebgc-tw/viewform?usp=sf_link
Hosted on Ausha. See ausha.co/privacy-policy for more information.
More from The Prompt Desk
In this one year anniversary episode of the podcast, your show hosts look back on what they have learned and observed over the past year. They made the following observations:
In this thought-provoking episode, AI experts Bradley Arsenault and Justin Macorin delve into the cutting-edge world of input validation using large language models (LLMs). The hosts explore how this innovative approach is transforming traditional data validation methods in software applications.
In this episode of Prompt Desk, hosts Bradley Arsenault and Justin Macorin dive into the evolving landscape of AI applications. They explore the shift from full automation dreams to the reality of human-AI collaboration, discussing why guided experiences are proving more effective than autonomous systems. The hosts share insights on designing AI interfaces, managing error rates, and the importance of human oversight in complex AI tasks. Whether you're an AI enthusiast, developer, or business leader, this episode offers valuable perspectives on harnessing the power of large language models while avoiding common pitfalls. Join Bradley and Justin as they unpack the challenges and opportunities in creating reliable, user-friendly AI applications that truly augment human capabilities.
In this episode of The Prompt Desk podcast, hosts Bradley Arsenault and Justin Macorin dive deep into the world of fine-tuning large language models. They discuss:
Have you ever tried to run a prompt in another language and gotten completely different results? It is a sad truth that our LLM models continue to work best in English. It is something that will probably always remain true because of the dominance of the English language, which ensures that a disproportionate amount of the worlds data is found in English.