01 / CODE / ENGINEERING
Code repositories & engineering tasks
Build reproducible engineering tasks around repositories, execution environments, and tests.
Task families include Terminal Bench, Frontier SWE, Coding Repo, Code Generation, and long-horizon agent work. A sample ties a pinned repository state to an issue, dependencies, tests, and an execution trace. Benchmark names describe task forms; they do not imply ownership of official data or affiliation.
DELIVERY / CONTENT
What a delivery can include
Defined with each project and its authorization scope.
- 01Authorized repository, pinned commit, and dependencies
- 02Task brief, tool permissions, and resource limits
- 03Reference solution, tests, grader, and execution trace
SCENARIO / EXAMPLE
A task in context
For a cross-module defect, an agent reproduces the failure, traces calls across files, patches the code, and runs independent tests. Longer tasks preserve intermediate decisions and rollbacks.
What we verify
Check licensing, pinned versions, reproducible builds, hidden-test separation, and leakage.