Leveraging E2E Test Context for LLM-Enhanced Test Data and Descriptions

Abstract

Automated software testing plays a critical role in improving software quality and reducing manual testing expenses. However, generating understandable and meaningful unit tests remains challenging, especially with frameworks optimized for coverage like Search-Based Software Testing (SBST). Large Language Models (LLMs) have the capability to generate human-like text, while capture/replay techniques can provide realistic data scenarios through trace logs, contributing to meaningful test case generation. This study introduces UTGen+, an approach that enhances LLM-based SBST by integrating trace logs from end-to-end tests, aiming to further improve test case understandability.
We conducted a comparative user study with 9 participants using UTGen+, original UTGen, and conventional SBST (EvoSuite), focusing on the effects of trace log inclusion on the naturalness and relevancy of comments, identifiers, and test data across several projects. The results indicated that while UTGen+ did not improve the naturalness and relevancy of comments and identifiers, it significantly enhanced the relevancy of test data. These findings suggest that incorporating contextual data can indeed benefit the generation of more relevant and understandable automated test cases.