Search Results

Background. Targeted unit tests, which aim to reveal a bug at a specific location in the program, are useful for various software development tasks, such as static analysis alarm validation, crash reproduction, or patch testing. While automated unit test generation has progressed significantly, existing approaches remain inefficient for targeted tests due to their reliance on manually-crafted heuristics. To address this shortcoming, Large Language Model (LLM) agents emerge as a potent solution that is free from predefined heuristics, owing to their superior reasoning and tool-use capabilities. Moreover, prior studies have shown that LLM agents can achieve comparable performance across several software engineering tasks with only minor adaptations. However, whether this adaptability extends to targeted unit test generation remains unexplored. Aims. This study aims to investigate the capabilities and limitations of LLM agents in targeted unit test generation under minimal custom setup. Method. We evaluate various combinations of frontier agent scaffolds and models on a benchmark featuring 210 real-world Java runtime exception bugs across 66 software projects. Specifically, LLM agents are tasked with generating a unit test suite that reproduce a target bug given only its location and the corresponding codebase. Results. Our evaluation shows that the top-performing combinations achieve a 89.5% success rate in target bug reproduction, surpassing the state-of-the-art unit test generation tools. Nevertheless, we identify recurring failure modes that illustrate current LLM agents' blind spots. Conclusions. These findings underscore the potential of LLM agents in targeted unit test generation, and provide valuable insights for further research in this area.