Fundamentals of Aligning General-Purpose AI

Abstract: The field of artificial intelligence (AI) is undergoing a paradigm shift, moving from neural networks trained for narrowly defined tasks (e.g., image classification and machine translation) to general-purpose models such as ChatGPT. These models are trained at unprecedented scales to perform a wide range of tasks, from providing travel recommendations to solving Olympiad-level math problems. As they are increasingly adopted in society, a central challenge is to ensure the alignment of general-purpose models with human preferences. In this talk, I will present a series of works that reveal fundamental pitfalls in existing alignment methods. In particular, I will show that they can: (1) suffer from a flat objective landscape that hinders optimization, and (2) fail to reliably increase the likelihood of generating preferred outputs, sometimes even causing the model to generate outputs with an opposite meaning. Beyond characterizing these pitfalls, our theory provides quantitative measures for identifying when they occur, suggests preventative guidelines, and has led to the development of new data selection and alignment algorithms, validated at large scale in real-world settings. Our contributions address both efficiency challenges and safety risks that may arise in the alignment process. I will conclude with an outlook on future directions, toward building a practical theory in the age of general-purpose AI.