GenAI For Creating Functional Tests for a REST API: A Case Study
Context
Generative AI is a revolutionary technology that has sparked immense excitement and research interest in the scientific community since the appearance of Large Language Models like GPT. Among the efforts and investments made to find beneficial usages of GenAI, some of them focus on how generative AI can improve the software development process through automatic code generations, bug detection, testing or deployment.
Motivation
The goal of this project is to explore the capabilities of some existing LLMs to automatically generate functional tests for an existing REST API. The API contains a swagger documentation that describes, using a .yaml file, all the existing endpoints of the API and their characteristics, like their parameters, their request body, the possible server’s responses and if the route can be accessed by authenticated users only. This file will be used as a starting point for the LLMs to generate and create the source code that perform the functional testing of the API.
Goal
Explore ongoing research and tools developed in automating the testing generation for REST APIs. Select 3 LLMs and compare their performance in terms of raw test coverage. Experiment different techniques like chain-of-thought reasoning’s prompts, fine-tuning to obtain an instruction-based model or function tools and compare their test coverage result between each other and the raw test coverage.
Major steps:
- Search and read existing papers on the domain.
- Select 3 LLMs and generate the functional tests with a single prompt. Assess their test coverage.
- Experiment with more advanced techniques and generate the functional tests with those techniques. Assess their test coverage.
- Write a report on the found results.
Requirements
Python and Node.js knowledge needed, as well as in REST APIs, swagger LLMs.
Pointers
Contact
Roman Macháček, roman.machacek@unibe.ch