STAREAST 2020 Concurrent Session : QA for ML: Testing around the AI Black Box


Thursday, May 7, 2020 - 11:15am to 12:15pm

QA for ML: Testing around the AI Black Box

There's been a lot of discussion around leveraging AI and machine learning (ML) for testing software, but seemingly less on how to test and provide QA for an AI- or ML-driven application. But Lauren Pehnke was faced with doing just that for a product that would combine natural language processing, computer vision, and image classification to drive ranked search results. She will outline the research on AI and ML she did before taking on this role, including basic vocabulary for the variants of AI involved and interesting AI "failures" to better inform what areas of the process would require a more critical eye. She will also cover testing approaches and strategies that have been useful in this space, borrowing from the scientific method and exploratory and black box testing, some ways to quantify qualitative evaluations of results when developing models, and a reminder that while the shiny new tech is cool, eventually it has to fit into an end-user experience. Come with your questions and your experiences to contribute to this conversational session so that we can all learn more about working with ML.

Lauren Pehnke

Lauren Pehnke's tech origin story really kicked off while earning her bachelor's in biotechnology and assisting some computer science majors with their indie game projects. She carried that same scientific curiosity and methodology forward through the years—performing visual and functional QA for Rock Band 4's multi-platform release; wrangling databases, vendors, and client support tickets for athenahealth's EHR system, and most recently starring as the QA specialist on back-office and machine learning projects at Aquent, an international staffing company.