Automating Unit Test Generation with Large Language Models: An Integrated Empirical and Theoretical Investigation

Abstract

Background: The emergence of large language models (LLMs) has introduced novel capabilities for program understanding and automated artifact generation, including unit tests. Recent empirical work suggests both promise and limitations of LLMs when applied to unit test generation tasks (Yang et al., 2024; Siddiq et al., 2024; Tang et al., 2024). However, existing studies vary in scope, metrics, and experimental controls, and there remains a need for an integrative study that synthesizes prior methodologies, aligns evaluation criteria with software engineering standards, and explores complementary approaches such as search-based software testing (SBST) and feedback-directed random testing (Pacheco et al., 2007; Harman & McMinn, 2010).

Objective: This study systematically investigates the effectiveness, reliability, and practical utility of LLM-based unit test generation compared with established automated testing techniques. We aim to provide a rigorous methodology, reproduceable evaluation framework, and nuanced theoretical interpretation to inform practitioners and researchers about when and how LLMs can augment or replace traditional test-generation approaches.

Methods: Drawing on methods and evaluation practices from prior literature on LLMs for software engineering (Fan et al., 2023; Hou et al., 2024; Rao et al., 2023), symbolic execution and loop characteristic analyses (Xiao et al., 2013), SBST (Harman & McMinn, 2010), and feedback-directed random test generation (Pacheco et al., 2007), we designed a controlled empirical study. The study uses a curated corpus of Java classes with documented behavior and existing high-quality JUnit suites, a set of LLM prompting strategies and model variants, and baseline automated tools. Evaluation metrics include functional correctness of generated tests, mutation score, coverage (statement/branch), fault-revealing power, human-readability, and maintenance cost proxies (Dustin et al., 2009; ISO/IEC/IEEE 24765:2017). We also present an extended analytical framework for interpreting results in light of testing taxonomies and automation frameworks (Mayeda & Andrews, 2021; Lonetti & Marchetti, 2018; Chandra et al., 2025).

Results: LLM-generated test suites exhibit notable strengths in producing human-readable, behaviorally-oriented unit tests that capture common usage patterns and edge-case assertions, often matching or exceeding baseline code-coverage for idiomatic code fragments (Yang et al., 2024; Siddiq et al., 2024). However, LLMs struggle on code segments dominated by complex loop constructs, intricate path conditions, and subtle numerical invariants—areas where symbolic execution and SBST demonstrate comparative advantages (Xiao et al., 2013; Harman & McMinn, 2010). Hybrid strategies that combine LLM-generated scaffolding with automated search or symbolic refinement substantially improve fault-revealing power and coverage plateau escape (Lemieux et al., 2023; Tang et al., 2024).

Conclusions: LLMs are a valuable addition to the automated testing toolkit but are not a universal replacement for established techniques. Best practices include using LLMs for rapid generation of semantically rich, human-readable tests and integrating them with SBST, symbolic execution, or feedback-directed random testing to improve coverage and fault detection. We conclude with a prescriptive automation framework, theoretical implications for the future of test automation, limitations of our study, and concrete directions for follow-up empirical and tooling work.