Room “Sala Seminari” - Abacus Building (U14)
Precise Language Models for Imprecise Humans:
Open Post-training Recipes and Verifiable Instruction Following
Speaker
Dr Valentina Pyatkin
Allen Institute for AI & University of Washington
Abstract
This talk examines methods for enhancing language model capabilities through post-training. While large language models have led to major breakthroughs in natural language processing, significant challenges persist due to the inherent ambiguity and underspecification in language. A crucial factor for successful human and AI interaction is the ability of language models or chatbots to follow human instructions precisely. A common feature of instructions are output constraints like ``only answer with yes or no" or ``mention the word `abrakadabra' at least 3 times" that the user adds to craft a more useful answer. Even today's strongest models struggle with fulfilling such constraints. I will show that most models strongly overfit on a small set of verifiable constraints from the benchmarks that test these abilities, a skill called precise instruction following, and are not able to generalize well to unseen output constraints. Further I will introduce a new benchmark, IFBench, to evaluate precise instruction following generalization on 58 new, diverse, and challenging verifiable out-of-domain constraints. In addition, I will show an analysis of how and on what data models can be trained to improve precise instruction following generalization with reinforcement learning from verifiable rewards. Throughout the presentation I will also touch upon more general post-training recipes, developed for Tülu and OLMo.
Short bio
Valentina Pyatkin is a postdoctoral researcher at the Allen Institute for AI and the University of Washington, advised by Prof. Hanna Hajishirzi. She is additionally supported by an Eric and Wendy Schmidt Postdoctoral Award. She obtained her PhD in Computer Science from the NLP lab at Bar Ilan University. Her work has been awarded an ACL Outstanding Paper Award and the ACL Best Theme Paper Award. During her doctoral studies, she conducted research internships at Google and the Allen Institute for AI, where she received the AI2 Outstanding Intern of the Year Award. She holds an MSc in AI from the University of Edinburgh and a BA from the University of Zurich. Valentina's research focuses on post-training and the adaptation of language models.
Contact person for the seminar: alessandro.raganato@unimib.it