Circular Image

A.E. Zaidman

info

Please Note

132 records found

Journal article (2026) - B.A. Ardıç, Q.T. Le Dilavrec, A.E. Zaidman
The integration of generative AI tools like ChatGPT into software engineering workflows opens up new opportunities to boost productivity in tasks such as unit test engineering. However, these AI-assisted workflows can also significantly alter the developer’s role, raising concerns about control, output quality, and learning, particularly for novice developers. This study investigates how novice software developers with foundational knowledge in software testing interact with generative AI for engineering unit tests. Our goal is to examine the strategies they use, how heavily they rely on generative AI, and the benefits and challenges they perceive when using generative AI-assisted approaches for test engineering. We conducted an observational study involving 12 undergraduate students who worked with generative AI for unit testing tasks, using ChatGPT running the GPT-3.5 model. We identified four interaction strategies, defined by whether the test idea or the test implementation originated from generative AI or from the participant. Additionally, we singled out prompting styles that focused on one-shot or iterative test generation, which often aligned with the broader interaction strategy. Students reported benefits including time-saving, reduced cognitive load, and support for test ideation, but also noted drawbacks such as diminished trust, test quality concerns, and lack of ownership. While strategy and prompting styles influenced workflow dynamics, they did not significantly affect test effectiveness or test code quality as measured by mutation score or test smells. ...

A socio-technical grounded theory

Journal article (2026) - Mark Swillus, Rashina Hoda, Andy Zaidman
Software testing crucial for ensuring software quality, yet developers’ engagement with it varies widely. Identifying the technical, organizational and social factors that lead to differences in engagement is required to remove barriers and utilize enablers for testing. While much research emphasizes the usefulness of software testing approaches and technical solutions, less is known about why developers do (not) test. This study investigates the first-hand experience of developers with software testing. The study illuminates how developers’ opinions about testing and their testing behavior changes. Through analysis of personal evolutions of practice, we explore when and why testing is used. Employing socio-technical grounded theory (STGT), we construct a theory by systematically analyzing data from 19 in-depth, semi-structured interviews with software developers. Allowing interviewees to reflect on how and why they approach software testing, we explore perspectives that are rooted in their contextual experiences. We develop eleven categories of circumstances that act as conditions for the application and adaptation of testing practices and introduce three concepts that we then use to present a theory of emerging testing strategies (ETS) that explains why developers do (not) use testing practices. This study reveals a new perspective on the connection between testing artifacts and collective reflection of practitioners, and it embraces testing as an experience in which human- and social aspects are entangled with organizational and technical circumstances. ...

A Report on Developer Experience with Type-Driven Development

Conference paper (2026) - Sára Juhošová, Andy Zaidman, Jesper Cockx
Static type systems are a well-established tool for preventing basic software errors, with more advanced ones providing strong guarantees of program correctness. Additionally, type systems encourage a "types first, implementation later"developer workflow known as "type-driven development"(TyDD). However, current widespread TyDD practices are based on simple type systems with limited expressivity, and the advanced tools being developed by researchers are not making it into mainstream programming languages. To determine how current practitioners experience the use of type-driven development and what inhibits its adoption by a wider range of developers, we conducted a survey with 130 participants from various backgrounds, asking them to describe their experience with current TyDD tools. According to them, TyDD can guide, communicate, and verify program implementation, but is currently limited by usability issues and missing features. Based on these results, we recommend that advanced TyDD tools be made available to a wider range of developers by investigating and addressing these limitations, with a focus on increasing expressivity while preserving usability. ...

An Empirical Study of Static and Dynamic Batching Strategies

Conference paper (2026) - Máté Oszkó, Xutong Liu, Andy Zaidman, Carolin Brandt
Continuous integration pipelines execute automated tests on every commit, consuming substantial energy. Batch testing, which groups multiple commits into a single test run, has been shown to reduce test executions in simulation studies, but no prior work has measured whether these reductions translate into actual energy savings. This study measures CPU package energy consumption of CI builds under a baseline run-all approach and two batching strategies (BatchStop4 and linear-4 lwd) across eight open-source Java projects. BatchStop4 achieves energy savings between 57% and 88%, while linear-4 lwd achieves savings between 59% and 94%. Energy savings correlate almost perfectly with time savings (r >0.99), indicating that batching reduces energy primarily by shortening total execution time. Baseline failure rate strongly predicts achievable savings, while CPU utilisation shows no significant relationship. These findings provide empirical evidence that batch testing effectively reduces the environmental footprint of continuous integration, particularly for projects with low failure rates. ...
Conference paper (2026) - Shujun Huang, Andy Zaidman, Sebastian Proksch
Continuous Integration (CI) has become a fundamental practice of modern software engineering, widely adopted in both commercial and open-source projects to enhance development efficiency and software quality. Existing empirical research has examined the relationship between the effectiveness of CI adoption and project characteristics, but it often assumes CI as a uniform intervention and neglects the heterogeneity of its effects. In reality, the CI effectiveness is shaped by contextual factors such as project maturity, domain, programming language, or development practices. Moreover, many prior studies rely on convenience samples of open-source projects, leaving the influence of dataset composition largely unexamined. This study employs quota sampling to construct a diversified dataset, ensuring representation across product characteristics and project activities. Building on this foundation, we examine the adoption of CI both at the aggregate level and within stratified, quotabased subsets of the dataset. The results show that the impact of CI is sensitive to different dataset compositions. When the contextual distribution of projects changes, both the direction and magnitude of CI's effects can shift, emphasizing that empirical evidence on CI effectiveness is inherently context-dependent. These findings indicate that the effectiveness of CI cannot be generalized without considering project heterogeneity. Future research should explicitly control dataset composition to better isolate or explain contextual effects. Besides, applying suitable sampling strategies can also minimize contextual bias, and help generate more robust and generalizable evidence for CI studies. ...
Conference paper (2026) - R. Arntzenius , X. Liu, A. Zaidman
Continuous integration is essential for software quality, yet the energy footprint associated with its frequent execution has largely remained invisible. We provide the first comprehensive baseline of CI energy use through a largescale study of 204 open-source Java projects with repeated measurements under Maven and Gradle. Our results show that energy use is highly skewed: while most projects consume energy modestly, a small number of “CI-intensive” systems can reach annual CI energy footprints of hundreds of kilowatt-hours, which is comparable to a quarter of an average EU household's electricity use. We further show that immediate, practical savings are possible: simply enabling dependency caching cuts energy by 30 % on average in some Maven projects and by over 90% in some Gradle cases. These findings matter not only for individual developers, but also for large organizations that run thousands of builds. In those settings, even small inefficiencies can add up to very large energy costs. By exposing where energy is consumed and how to reduce it, our study establishes an actionable foundation for greener CI pipelines. ...
Foreword postscript (2026) - Breno Miranda, Ana Paiva, Markus Borg, Andy Zaidman

An Empirical Study of SpotBugs Effort Levels

Static analysis tools are commonly used in continuous integration (CI) pipelines to detect potential defects without executing the program. Such tools offer configuration options that control how thoroughly the code is analyzed. In SpotBugs, this is done through an effort level setting (Min, Less, More, Max), where higher levels enable deeper and more computationally intensive analysis. In this paper, we study how these effort levels affect energy consumption and analysis results. Our results show that higher effort levels generally consume more energy, which fits our intuition. However, the increase is not uniform: deeper static analysis comes with additional energy cost, but the marginal benefit of higher effort levels may be limited in some cases. This suggests that static analysis tool designers could make energy consumption more transparent and provide configuration options that explicitly expose trade-offs between analysis depth and energy cost. ...
Conference paper (2026) - Andre Hora, Andy Zaidman
Best testing practices state that tests should verify a single functionality or behavior of the system. Tests that verify multiple behaviors are harder to understand, lack focus, and are more coupled to the production code. An attempt to identify this issue is the test smell Eager Test, which aims to capture tests that verify too much functionality based on the number of production method calls. Unfortunately, prior research suggests that counting production method calls is an inaccurate measure, as these calls do not reliably serve as a proxy for functionality. We envision a complementary solution based on runtime analysis: we hypothesize that some tests that verify multiple behaviors will likely cover multiple paths of the same production methods. Thus, we propose a novel test smell named Test Obsessed by Method, a test method that covers multiple paths of a single production method. We provide an initial empirical study to explore the presence of this smell in 2,054 tests provided by 12 test suites of the Python Standard Library. (1) We detect 44 Tests Obsessed by Methods in 11 of the 12 test suites. (2) Each smelly test verifies a median of two behaviors of the production method. (3) The 44 smelly tests could be split into 118 novel tests. (4) 23% of the smelly tests have code comments recognizing that distinct behaviors are being tested. We conclude by discussing benefits, limitations, and further research. ...
Conference paper (2026) - Xutong Liu, Andy Zaidman
Software testing is essential to ensure the reliability of software. To that end, test suites are often executed repeatedly, due to frequent developer validation runs, and practices like continuous integration. This repeated execution leads to substantial energy consumption and thus potential environmental impact. In this paper, we conduct an exploratory study to examine whether current Large Language Models (LLMs) are capable of reengineering unit tests to improve their energy efficiency while maintaining test effectiveness. Our results indicate that, using a straightforward prompt, only a small subset of unit tests shows improved energy efficiency, with negligible impact on test effectiveness. Subsequently, we scrutinize our results for potential reasons for the relatively low number of energy-reengineered unit tests. We observe that energy efficiency related information is scarce on prominent platforms like StackOverflow and GitHub. This scarcity of information potentially affects how LLMs can be trained on the topic of energy efficiency. ...
Software testing is essential, and automatic test case generation can be an important aid to software engineers. However, generated tests are sometimes difficult to understand. Test summarization approaches that provide an overview of what exactly is tested can provide help, but existing summarization approaches generate documentation that is lengthy and redundant. In this paper, we investigate whether large language models (LLMs) can be used to generate more concise, yet understandable summaries. In a small-scale user study with 11 participants, we obtained positive feedback on the LLM-generated summaries. ...
Automated unit test generators, particularly searchbased software testing tools like EvoSuite, are capable of generating tests with high coverage. Although these generators alleviate the burden of writing unit tests, they often pose challenges for software engineers in terms of understanding the generated tests. To address this, we introduce UTGen, which combines searchbased software testing and large language models to enhance the understandability of automatically generated test cases. We achieve this enhancement through contextualizing test data, improving identifier naming, and adding descriptive comments. Through a controlled experiment with 32 participants from both academia and industry, we investigate how the understandability of unit tests affects a software engineer's ability to perform bug-fixing tasks. We selected bug-fixing to simulate a real-world scenario that emphasizes the importance of understandable test cases. We observe that participants working on assignments with UTGen test cases fix up to 33% more bugs and use up to 20% less time when compared to baseline test cases. From the post-test questionnaire, we gathered that participants found that enhanced test names, test data, and variable names improved their bugfixing process. ...
Conference paper (2025) - Sára Juhošová, Andy Zaidman, Jesper Cockx
Interactive theorem provers (ITPs) are programming languages which allow users to reason about and verify their programs. Although they promise strong correctness guarantees and expressive type annotations which can act as code summaries, they tend to have a steep learning curve and poor usability. Unfortunately, there is only a vague understanding of the underlying causes for these problems within the research community. To pinpoint the exact usability bottlenecks of ITPs, we conducted an online survey among 41 computer science bachelor students, asking them to reflect on the experience of learning to use the Agda ITP and to list the obstacles they faced during the process. Qualitative analysis of the responses revealed confusion among the participants about the role of ITPs within software development processes as well as design choices and tool deficiencies which do not provide an adequate level of support to ITP users. To make ITPs more accessible to new users, we recommend that ITP designers look beyond the language itself and also consider its wider contexts of tooling, developer environments, and larger software development processes. ...

An industrial case study in language engineering with Spoofax

Journal article (2025) - Olav Bunte, Jasper Denkers, Louis C.M. van Gool, Jurgen J. Vinju, Eelco Visser, Tim A.C. Willemse, Andy Zaidman
Domain-specific languages (DSLs) promise to improve the software engineering process, e.g., by reducing software development and maintenance effort and by improving communication, and are therefore seeing increased use in industry. To support the creation and deployment of DSLs, language workbenches have been developed. However, little is published about the actual added value of a language workbench in an industrial setting, compared to not using a language workbench. In this paper, we evaluate the productivity of using the Spoofax language workbench by comparing two implementations of an industrial DSL, one in Spoofax and one in Python, that already existed before the evaluation. The subject is the Open Interaction Language (OIL): a complex DSL for implementing control software with requirements imposed by its industrial context at Canon Production Printing. Our findings indicate that it is more productive to implement OIL using Spoofax compared to using Python, especially if editor services are desired. Although Spoofax was sufficient to implement OIL, we find that Spoofax should especially improve on practical aspects to increase its adoptability in industry. ...

Incrementally Mining Large Source Code Repositories

Conference paper (2025) - Quentin Le Dilavrec, Andy Zaidman
Modern software systems are large, with a project like Chromium reaching more than 30 million lines of code. Analyzing these large-scale projects over multiple versions rapidly becomes very expensive, and creating tools that can work at this scale is a challenge. This paper presents the HyperAST approach, that exploits the locality and redundancy of source code, to maintain thousands of Syntax Tree (AST) versions in memory. In particular, we contribute a programmatic interface to HyperAST that helps define the incremental computation of code metrics and efficient explorations of the fine-grained abstract syntax representation of source code. ...

A systematic mapping study of qualitative methods

Software testing research has provided metrics on efficiency, error rates, and insights into the effectiveness of testing methodologies and tools. However, these tell only a part of the story. The qualitative dimension, which studies experiences, perceptions, and decision-making processes is crucial, but less prevalent in literature. This study aims to systematically map qualitative research in software testing to consolidate and categorize the methodologies used in qualitative testing research, highlight their importance, and identify patterns, gaps, and future directions. We conducted a systematic mapping study, identifying and analyzing 102 primary studies from 2003 to 2023. We categorized the studies according to research strategies, data collection, and data analysis methods. We identified case studies and grounded theory as the most prevalent research strategies. Researchers primarily used semi-structured interviews and thematic analysis to understand how practitioners work and gather stakeholder perspectives. The subject areas most covered by qualitative studies included software testing processes and risks, and test automation. Areas such as test oracles, and machine learning were underrepresented. We also assessed the quality of reporting and the methodological rigor, emphasizing the challenges and limitations identified during the process. Through this study, we provide a comprehensive overview of qualitative research practices in software testing, revealing trends, gaps, and methodological insights. ...

Mining Meaningful Code Review Orders From GitHub

Conference paper (2025) - A. Bouraffa, C.E. Brandt, A.E. Zaidman, W. Maalej
Developers use tools such as GitHub pull requests to review code, discuss proposed changes, and request modifications. While changed files are commonly presented in alphabetical order, this does not necessarily coincide with the reviewer's preferred navigation sequence. This study investigates the different navigation orders developers follow while commenting on changes submitted in pull requests. We mined code review comments from 23,241 pull requests in 100 popular Java and Python repositories on GitHub to analyze the order in which the reviewers commented on the submitted changes. Our analysis shows that for 44.6% of pull requests, the reviewers comment in a non-alphabetical order. Among these pull requests, we identified traces of alternative meaningful orders: 20.6% (2,134) followed a largest-diff first order, 17.6% (1,827) were commented in the order of the files' similarity to the pull request's title and description, and 29% (1,188) of pull requests containing changes to both production and test files adhered to a test-first order. We also observed that the proportion of reviewed files to total submitted files was significantly higher in non-alphabetically ordered reviews, which also received slightly fewer approvals from reviewers, on average. Our findings highlight the need for additional support during code reviews, particularly for larger pull requests, where reviewers are more likely to adopt complex strategies rather than following a single predefined order. ...
Conference paper (2024) - Khalid El Haji, Carolin Brandt, Andy Zaidman
Writing unit tests is a crucial task in software development, but it is also recognized as a time-consuming and tedious task. As such, numerous test generation approaches have been proposed and investigated. However, most of these test generation tools produce tests that are typically difficult to understand. Recently, Large Language Models (LLMs) have shown promising results in generating source code and supporting software engineering tasks. As such, we investigate the usability of tests generated by GitHub Copilot, a proprietary closed-source code generation tool that uses an LLM. We evaluate GitHub Copilot's test generation abilities both within and without an existing test suite, and we study the impact of different code commenting strategies on test generations.Our investigation evaluates the usability of 290 tests generated by GitHub Copilot for 53 sampled tests from open source projects. Our findings highlight that within an existing test suite, approximately 45.28% of the tests generated by Copilot are passing tests; 54.72% of generated tests are failing, broken, or empty tests. Furthermore, if we generate tests using Copilot without an existing test suite in place, we observe that 92.45% of the tests are failing, broken, or empty tests. Additionally, we study how test method comments influence the usability of test generations. ...

An Investigation into Why Software Engineers (Occasionally) Ignore Coverage Checks

Conference paper (2024) - Alexander Sterk , Mairieli Wessel, Eli Hooten, Andy Zaidman
Many modern code coverage tools track and report code coverage data generated from running tests during continuous integration. They report code coverage data through a variety of channels, including email, Slack, Mattermost, or through the web interface of social coding platforms such as GitHub. In fact, this ensemble of tools can be configured in such a way that the software engineer gets a failing status check when code coverage drops below a certain threshold. In this study, we broadly investigate the opinions and experience with code coverage tools through a survey among 279 software engineers whose projects use the Codecov coverage tool and bot. In particular, we are investigating why software engineers would ignore a failing status check caused by drop in code coverage. We observe that >80% of software engineers-at least sometimes-ignore these failing status checks, and we get insights into the main reasons why software engineers ignore these checks. ...

How Developers Like Their Amplified Tests

Journal article (2024) - Carolin Brandt, Ali Khatami, Mairieli Wessel, Andy Zaidman
Test amplification makes systematic changes to existing, manually written tests to provide tests complementary to an automated test suite. We consider developer-centric test amplification, where the developer explores, judges and edits the amplified tests before adding them to their maintained test suite. However, it is as yet unclear which kind of selection and editing steps developers take before including an amplified test into the test suite. In this paper we conduct an open source contribution study, amplifying tests of open source Java projects from GitHub. We report which deficiencies we observe in the amplified tests while manually filtering and editing them to open 39 pull requests with amplified tests. We present a detailed analysis of the maintainer's feedback regarding proposed changes, requested information, and expressed judgment. Our observations provide a basis for practitioners to take an informed decision on whether to adopt developer-centric test amplification. As several of the edits we observe are based on the developer's understanding of the amplified test, we conjecture that developer-centric test amplification should invest in supporting the developer to understand the amplified tests. ...