Abstract
Language models can generate programming exercises and solutions, but incorrect answers, weak tests, and repetitive tasks can limit their value as training data. This thesis builds on CodifieRL to generate synthetic coding tasks from real repositories and investigate whether execution-based filtering improves their quality and usefulness for training code models.
Supervision
- Roman Machacek
Motivation
Real code provides seeds for realistic programming tasks. CodifieRL already supports synthesizing instructions, solutions, and executable tests from these seeds. However, a generated solution passing its own generated tests does not guarantee a useful training example. The challenge is to produce diverse tasks with meaningful tests at an affordable generation cost.
Goal
The student will develop and evaluate a synthetic data generation workflow using CodifieRL. Expected deliverables include:
- a reproducible dataset of programming tasks, reference solutions, and tests derived from Python repositories
- comparisons of generation with and without execution-based filtering, including checks for duplicates and weak tests
- measurements of task diversity, test strength using deliberately faulty solutions, and generation cost per accepted example
- a controlled fine-tuning study comparing equal-size datasets, evaluating code generation on independent held-out tasks and checking for overlap with training data
Requirements
Strong Python and machine learning skills, with experience in PyTorch. Familiarity with language models, software testing, and Git is expected; experience with Docker and model fine-tuning is helpful.
Pointers
- Y. Wei et al., “Magicoder: Empowering Code Generation with OSS-Instruct,” ICML, 2024.
- W. U. Ahmad et al., “OpenCodeInstruct: A Large-scale Instruction Tuning Dataset for Code LLMs,” 2025.