Use accepted, reviewable outcomes—not generated lines of code—as the main unit.
REPRODUCIBLE EVALUATION · REVIEWED 2026-08-05
MonkeyCode pilot research methodology.
This protocol helps teams produce their own auditable evidence. It does not claim that MonkeyCode has achieved any performance, quality, or cost result.
Protocol
Define the test before running the tool.
Keep rejected attempts and missing measurements. Removing failures or turning unknown values into zero makes comparisons misleading.
- 01Select representative work.
Choose at least three bounded tasks from a fixed repository commit: a defect, a small feature, and a test or documentation change. Exclude sensitive repositories from public datasets.
- 02Freeze acceptance criteria.
Record required tests, build checks, security constraints, files allowed to change, and the reviewer decision rule before the first run.
- 03Control the comparison.
Use the same task definitions and repository baseline for each workflow. Record tool version, model, endpoint, environment, permissions, and material prompt changes.
- 04Record every attempt.
Measure wall-clock time, active review, retries, accepted outcome, model usage, compute cost, and failure category. One row represents one attempt.
- 05Publish with limits.
Remove secrets and proprietary content, version the dataset, disclose exclusions and missing fields, and report distributions and sample counts—not unsupported universal claims.
Contribute a real case
Turn a completed pilot into auditable evidence.
Use the case template to disclose the evaluation window, sample size, baseline, decision, limitations, evidence URL, and publication consent. Do not identify a customer without approval.
Data dictionary
Fields in the blank pilot-runs template.
The downloadable file contains only headers. Blank means not measured or unavailable; zero means a measured value of zero.
| Field | Definition |
|---|---|
| task_id / run_id | Stable task identifier and run number. Use one row per attempt. |
| repository_commit | Exact baseline commit SHA used for the run. |
| monkeycode_version | Release, image tag, or source commit under evaluation. |
| model_id / endpoint_type | Exact model identifier and one of provider, gateway, or local. |
| run_started_at_utc | ISO 8601 UTC timestamp, for example 2026-07-15T10:00:00Z. |
| accepted | true only when the predefined acceptance criteria and review gate pass. |
| retries | Number of retries within this recorded attempt; do not count separate run rows. |
| elapsed_seconds / review_minutes | Wall-clock attempt duration and active human review time. |
| input_tokens / output_tokens | Provider-reported token counts; leave blank when unavailable. |
| model_cost / compute_cost / currency | Measured costs for this attempt; leave unknown values blank rather than writing zero. |
| failure_category | One of none, setup, environment, model, tool, test, review, policy, or unknown. |
| notes | Sanitized context needed to interpret the run; never include source, prompts, secrets, or personal data. |
Interpretation limits
A pilot supports a local decision, not a universal leaderboard.
Repository complexity, task selection, model changes, environment configuration, reviewer skill, and network conditions all affect outcomes. Repeat material changes and retain the raw rows behind every summary.
Security, policy, or reliability failures can block rollout even when averages look favorable.
This site provides a protocol and blank template. It does not publish fabricated benchmark data.