Last month I wrote about the six questions I ask every new model. Those questions are good for a first impression, but they have a blind spot: they tell me how a model talks about code, not whether it can survive contact with my code. So I built a small, ugly, reproducible harness that runs any coding model against five tasks drawn from my own git history, scores the results with my real test suite, and writes the outcome to a CSV. This post is that harness, the reasoning behind it, and where free-tier tooling fits without falling apart.

Why repo-derived tasks beat benchmark tasks

Public benchmarks (HumanEval-style) have two problems for day-to-day tool selection. First, they're saturated and contaminated — models have seen them. Second, they don't look like your work. My work is: small refactors in a Python service, tests that fail for boring reasons, and occasional SQL migrations. So instead of asking "is this model smart," I ask "does this model reduce the time I spend on the five task shapes I actually do."

The trick that makes this cheap: your git history is a labeled dataset. Every commit that fixed a bug is a task (reproduce the fix) with a built-in grader (the tests that existed at the parent commit, plus the test added in the fix commit, if any).

The harness

Step 1: mine five candidate tasks from git.

# Find bugfix-ish commits that also touched test files
for sha in $(git log --oneline -300 --grep='fix' -i --format='%h'); do
  if git show --name-only --format='' $sha | grep -q 'test'; then
    echo "$sha $(git log -1 --format='%s' $sha)"
  fi
done | head -20

I pick five where the diff is under 60 lines. Small diffs keep the test honest — you're measuring one decision, not endurance.

Step 2: for each chosen commit, build a task directory.

#!/usr/bin/env bash
# make_task.sh <fix_commit_sha> <task_name>
set -euo pipefail
SHA=$1; NAME=$2
mkdir -p "tasks/$NAME"
git worktree add "tasks/$NAME/repo" "${SHA}^" 2>/dev/null
# The 'gold' patch and the grading test come from the fix commit itself
git show "$SHA" -- . ':!*test*' > "tasks/$NAME/gold.patch"
git show "$SHA" -- '*test*' > "tasks/$NAME/grading_test.patch"
# Prompt = commit message of the fix, i.e. the problem statement
git log -1 --format='%B' "$SHA" > "tasks/$NAME/prompt.txt"

Now each task has: a repo checked out before the fix, a problem statement (the original commit message), and a grader (the test that arrived with the fix). Nothing is hand-written.

Step 3: run the model and grade.

# grade.py — run from tasks/<name>/repo after applying the model's patch
import subprocess, json, sys, pathlib

def run(cmd):
    return subprocess.run(cmd, shell=True, capture_output=True, text=True)

result = {"applies_clean": False, "tests_pass": False, "overreach": False}

patch = pathlib.Path("../model.patch")
if run(f"git apply --check {patch}").returncode == 0:
    result["applies_clean"] = True
    run(f"git apply {patch}")
    run("git apply ../grading_test.patch")  # the hidden test
    t = run("python -m pytest -x -q")
    result["tests_pass"] = (t.returncode == 0)
    # Overreach: did the model edit files outside the gold patch's scope?
    gold_files = set(run("git apply --numstat ../gold.patch").stdout.split())
    changed = run("git diff --name-only").stdout.split()
    result["overreach"] = len(set(changed) - set(gold_files)) > 1

print(json.dumps(result))

Three scores per task, five tasks, fifteen data points. That's it. I deliberately score overreach because in my experience the most expensive model failure isn't a wrong answer — it's a confident model quietly reformatting three unrelated files in the same patch.

Scoring table from my first run

I ran this against two setups: my usual paid tool, and a free model served through MonkeyCode, which currently offers free model access plus a free server option for exactly this kind of side experiment. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I picked it for the comparison run because the experiment's entire premise is "is free good enough," and a zero-cost server meant I could rerun the harness as many times as I wanted without watching a meter. I'm not going to quote numbers as if they're universal — your repo is not my repo — but the shape of the result was consistent:

Two honest observations. First, the free model was fine on the unambiguous tasks — which, embarrassingly, are most of my real tickets. Second, both setups failed the ambiguous-prompt task, which confirmed the harness is measuring task clarity as much as model quality. A task neither setup can solve is a bad task, not a bad model; I rewrote that prompt and both passed.

Limitations, plainly

• n=5 is a smoke test, not a benchmark. It tells you where to dig, not what to buy.
• The grader only sees tests that existed. A model can pass the hidden test and still write worse code than the gold patch. I skim every diff by hand; the CSV just triages.
• Contamination risk cuts both ways. If your repo is public and popular, models may have seen your fix commits. This works best on private or unglamorous codebases — which is most of ours.
• Free tiers move. "Free models and a free server" is an availability claim as of writing, not a promise about quotas, hardware, or permanence. Don't build CI on it; treat it as a lab bench.
• Don't use this for security-sensitive patches. Grading is "does the test pass," not "is the patch safe."

Who this is for (and who should skip it)

Use this if you're evaluating coding assistants and you're tired of demo-driven decisions. Skip it if your work is greenfield-heavy — the whole method depends on having a fix-history to mine — or if you need statistically meaningful numbers for a procurement decision, in which case you need a bigger labeled set and more discipline than a bash script gives you.

What changed for me: the question stopped being "which model is best" and became "which tasks am I still doing by hand that a free model already handles." The harness took an afternoon; the answer paid for it in a week. If you want to replicate it, the three scripts above are the whole thing — point them at your own history, and if you need somewhere free to run the candidate model, that's a reasonable first use of MonkeyCode's free tier before you commit to anything.

I'd be curious what task shapes other people's git histories turn up — mine were depressingly dominated by off-by-one errors.