Show Notes
Lots of people like to talk about the importance of prompts, context, and what is sent to an LLM. Few discuss the even more important aspect of an LLM-driven system in evaluating its output. In this episode, we discuss traditional and modern metrics used to evaluate LLM outputs. And, we review the common frameworks for obtaining that feedback. Though evals are a lot of work (and easy to do poorly), those building (or buying) LLM-driven systems should be transparent about their process and the current state of their eval framework.
More from The Behavioral Data Science Podcast
LLMs are one type of generative artificial intelligence (AI) which is one type of AI/machine learning which is one type of approach to computational modeling. In this episode Jake and David talk about the ethics of computational modeling, generally, and how behavior scientists might incorporate these ideas as they begin to build models specific to their everyday work settings.
In this episode, Jake and David discuss the Stanford AI Index Report (https://hai.stanford.edu/ai-index) with an eye toward the intersection of ethics, AI, and behavior science.
The term "bias" means different things to people coming from different scientific backgrounds. In this episode, Jake and David discuss those different definitions of bias, how those definitions impact the conversation around bias in AI, and recommendations for mitigating "bias" in AI systems, regardless of the point of view from which you come.
In this episode, Jake and David discuss the unique ethical concerns raised by LLMs, generative AI, agentic systems, and multi-modal AI.
In this episode, Jake and David discuss the ever-present possibility of bias in AI systems, and the resulting work and implications that has to mitigate AI bias when using LLM-driven products.