Towards More Transparent AI: Consistent Explanations and Interpretable Summaries
יום רביעי 02.12 11:00 - 12:00
- Graduate Student Seminar
-
LIB03
Abstract: As AI agents become increasingly capable and autonomous, understanding their behavior and decisions becomes essential. Yet, this remains challenging. Some agents, such as those trained through reinforcement learning, cannot explain their behavior, leaving users to infer their strategies from observations. Others, such as large language models, can generate convincing natural-language explanations, but these explanations do not necessarily reflect how their decisions were made. In this talk, I will explore these challenges through research on reinforcement learning agents and large language models. I will present methods for generating interpretable, natural-language summaries of agent behavior, and examine the reliability of explanations produced by language models. In particular, I will discuss how we can measure and improve the consistency between a model's decisions and its explanations, and investigate whether models stand behind the claims they make when explaining their answers. Together, this work highlights the challenges of understanding learned agents and the importance of developing explanations that are not only accessible to humans, but also consistent with the behavior they aim to describe.