Context
AI coding assistants are now widely used in professional software development. The prompts developers write, the responses they receive, the subsequent edits they make, and the rework they perform before committing code remain almost entirely invisible for in-depth analysis. Without access to the full interaction between developer intent (i.e., a prompt) and AI output, it is not possible to study whether AI-generated code actually addresses what the developer needed.
Motivation
Most AI coding assistants communicate with model APIs that are not directly observable. A proxy-based approach — where developer tools route their large language model API calls through a logging server — makes this data accessible without requiring developers to change their tools or workflows. Because most AI coding tools support the OpenAI API format and allow configuring a custom endpoint, a single proxy server can instrument a wide range of tools with minimal setup. An IDE extension running alongside the proxy captures what the proxy cannot: how the developer interacts with generated code in the editor, file state at key checkpoints, and version control anchors.
Goal
The goal of this project is to design, implement, and pilot this two-component instrumentation infrastructure with a small group of consenting developer participants.
The outcome of this project includes:
- A server-side API proxy implementing the OpenAI chat completions protocol, including streaming support, that intercepts, logs, and forwards model API calls transparently.
- A lightweight Visual Studio Code extension that captures editor-side events linked to AI interactions: code state at save and commit checkpoints, developer actions on generated code, git provenance, and lightweight in-the-moment developer feedback.
- A linking mechanism correlating proxy-captured AI exchanges with editor-side events, enabling reconstruction of the full prompt → response → developer action chain per interaction.
- A pilot deployment with a small group of consenting participants producing an initial dataset and a feasibility report characterizing what the infrastructure can and cannot reliably capture.
Requirements
- Programming experience in TypeScript or JavaScript and Python or Node.js
- Familiarity with web APIs and HTTP; experience with streaming APIs is a plus
- Basic understanding of Git and version control
- Interest in empirical software engineering research
Contact
- Primary Supervision: Dr. Nitish Patkar