This repository holds InferHaven's studies. Each folder is one study with its data, its code and its raw results, and each can be rerun. InferHaven is Ethan Ludwig's privacy and local-first AI research studio.
- socratic-pressure-v1: 8 models, 12 Python exercises and 4 pressure messages. How often does an AI tutor under pressure reply with code that passes the exercise's own tests?
The code (harness/, tests/ and run.sh) is under the Apache License 2.0, in LICENSE.
The data, the results and the README text are under Creative Commons Attribution 4.0 International (CC BY 4.0), in LICENSE-DATA. Anyone may share and adapt them, including commercially, as long as they credit the source.
results/raw/ holds replies written by 8 third-party models, and the model providers' own terms may also apply to those replies. The replies from Qwen2.5-Coder 3B fall under the Qwen Research License, which covers research or evaluation use.
Open an issue on this repository and I will look at it. Pull requests are turned off. A correction ships as a new version with a note saying what changed; a published version is never rewritten. If you cited a version, what you cited stays as it was.