Hamel Husain is a ML Engineer with over 20 years of experience. He has worked with innovative companies such as Airbnb and GitHub, which included early LLM research used by OpenAI, for code understanding. He has also led and contributed to numerous popular open-source machine-learning tools. Hamel is currently an independent consultant helping companies build AI products.
Title: Stop Guessing, Start Learning: A Practical Guide to Building AI That Works
Abstract: We’ve all experienced it: an AI feature that works perfectly in a demo but fails in unexpected ways with real users. The common response is to tweak the prompt and hope for the best. This keynote offers a more reliable path forward, grounded in a simple, repeatable process: Analyze, Measure, and Improve. We will reframe AI evaluation not as a final step, but as the core learning engine for building better products.
The journey begins with the Analyze phase, where we move from assumption to understanding. Instead of guessing what might be wrong, we will show how to systematically review your AI’s behavior to gain a deep, qualitative understanding of its actual failure modes. This creates a clear, evidence-based picture of what needs to be fixed.
Next, we Measure what matters. This phase is about translating those qualitative insights into reliable, quantitative metrics. By developing custom, application-specific checks, you can create a clear scoreboard to track progress, prevent regressions, and make data-driven decisions about quality.
Finally, this data empowers the Improve phase. With a clear view of what’s broken and how often, you can make targeted interventions that lead to real, measurable improvements in your AI’s performance and reliability. This cycle creates a powerful feedback loop, turning evaluation into a continuous learning process.
Dr Molenaar holds Master’s degrees in Cognitive Psychology and International Business studies and a PhD in Educational Sciences (University of Amsterdam).