Room “Sala Seminari” - Abacus Building (U14)
LLMs as Test Oracle Generators:
Evaluating and Advancing Their Role in Software Testing
Speaker
Valerio Terragni
University of Auckland, New Zealand
Abstract
The oracle problem is the problem of determining whether a program behaves correctly. It remains a core challenge in automated software testing. While modern test generation tools are good at generating high-coverage test cases, their oracles often merely reflect implemented behavior, making them suitable only for regression testing. Large Language Models (LLMs) present a promising direction for overcoming this limitation by generating oracles that could predicate on expected rather than implement behavior.
This talk explores the emerging role of LLMs in oracle generation through two complementary studies. First, we present an empirical evaluation of LLMs' effectiveness in generating fault-revealing assertions, examining how prompting strategies and context size influence the quality and correctness of generated oracles. Second, we introduce LLMProphet, a novel black-box technique that leverages few-shot prompting and leave-one-out validation to identify incorrect assertions in regression test suites. By treating existing tests as contextual examples, LLMProphet detects misaligned behavior without access to source code.
Together, these studies offer insights into the capabilities and limitations of LLM-driven oracle generation, opening new directions for test automation and fault detection.
Short Bio:
Valerio is a Senior Lecturer (equivalent to Associate Professor in European universities) and also the Program Director of the Bachelor of Software Engineering at The University of Auckland, New Zealand. From 2017 to 2020, he was a Senior Research Fellow at USI in Lugano, Switzerland. In 2017, he obtained his PhD at The Hong Kong University of Science and Technology (HKUST) in Hong Kong. He received his B.Sc. and M.Sc. degrees in Computer Science from The University of Milano-BICOCCA, Italy. His research interest is mainly software testing, he has been published in top software engineering venues, such as TSE, TOSEM, ICSE, ESEC/FSE, ASE, ISSTA, among others.
contact person for this Seminar: prof. Leonardo Mariani