Most model comparisons I read have the same flaw: the models never touch the same problem under the same conditions. One gets a carefully groomed prompt, the other gets a sloppy one. One runs against a clean checkout, the other inherits half-finished edits from the previous attempt. The conclusion tells you more about the harness than the models.

I wanted a comparison I could actually trust for my codebase, so I built a small bake-off harness around git worktree. It runs two coding models against the identical task, in identical isolated checkouts, and reduces the result to a table: did the tests pass, how big was the diff, did it touch files it was told not to touch. This post is that harness, plus the judgment calls that make the results meaningful.

This pairs naturally with a free tier: a bake-off is bursty, short-lived work, so it is a good fit for zero-cost compute. I ran mine using MonkeyCode's free model access on their free server option, which meant the whole experiment cost nothing and left nothing running afterward. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The harness itself is plain git, shell, and your test runner — swap in any two models or agents and it works the same.

Why worktrees instead of branches or clones

The isolation requirement is strict: both models must see byte-identical starting state, and neither run may contaminate the other or your working copy. Three options:

Worktrees win: each run gets its own directory backed by the same object store, your main checkout is never touched, and cleanup is one command. The one sharp edge — two worktrees cannot have the same branch checked out — is actually a feature here, since it forces each run onto its own throwaway branch.

The harness

#!/usr/bin/env bash
# bakeoff.sh <task-file> <model-a-cmd> <model-b-cmd>
# Runs both agents on the same task in isolated worktrees.
set -euo pipefail

TASK_FILE="$1"; CMD_A="$2"; CMD_B="$3"
BASE="$(git rev-parse HEAD)"
STAMP="$(date +%Y%m%d-%H%M%S)"

run_leg() {
  local name="$1" cmd="$2"
  local dir="../bakeoff-${name}-${STAMP}"
  git worktree add --detach "$dir" "$BASE" >/dev/null
  cp "$TASK_FILE" "$dir/TASK.md"
  (
    cd "$dir"
    # Baseline: tests must pass BEFORE the agent runs, else the run is invalid.
    if ! ./run-tests.sh >/dev/null 2>&1; then
      echo "BASELINE_FAIL"; exit 0
    fi
    timeout 1800 bash -c "$cmd" </dev/null >/dev/null 2>&1 || true
    ./run-tests.sh >/dev/null 2>&1 && echo "TESTS_PASS" || echo "TESTS_FAIL"
  )
}

A_RESULT="$(run_leg a "$CMD_A")"
B_RESULT="$(run_leg b "$CMD_B")"

report() {
  local name="$1" result="$2"
  local dir="../bakeoff-${name}-${STAMP}"
  if [ "$result" = "BASELINE_FAIL" ]; then
    echo "| $name | INVALID (baseline red) | — | — |"
    return
  fi
  local stat scope
  stat="$(git -C "$dir" diff --shortstat "$BASE")"
  scope="$(git -C "$dir" diff --name-only "$BASE" | grep -cvE '^(src|tests)/' || true)"
  echo "| $name | $result | $stat | $scope out-of-scope files |"
}

echo "| leg | tests | diff size | scope violations |"
echo "|---|---|---|---|"
report a "$A_RESULT"
report b "$B_RESULT"

run-tests.sh is whatever your project already uses, wrapped to exit non-zero on failure. TASK.md is the single prompt both models receive — write it once, and resist editing it between legs.

The three details that make or break the comparison

1. The baseline gate. The most common way to fool yourself is handing a model a repo whose tests are already red, then crediting or blaming it for the outcome. The harness runs the test suite before the agent does anything; a red baseline invalidates the leg. In my runs this fired more often than I expected — usually a fixture that depended on state from my main working copy.

2. Scope violations as a first-class metric. A model that fixes the bug but also "helpfully" reformats your config files or bumps dependencies is producing review burden, not value. The report counts changed files outside the directories the task permits. When two models both pass tests, this column is usually what decides it for me.

3. One task is an anecdote, not a result. A single bake-off tells you almost nothing; models have real variance run-to-run. I treat one execution as a data point and only draw conclusions after the same pairing across four or five distinct tasks (a bugfix, a small feature, a refactor, a test-writing task). If that sounds like a lot of compute: this is exactly where the free access matters. Each leg is a few minutes of burst work, and running the whole matrix on MonkeyCode's free models and free server meant I could repeat runs without rationing attempts or leaving a bill running. If you want to try the same setup, their free tier is a reasonable place to start — but the harness works with whatever two agents you have.

How to read the output

My decision rules, refined over a few weeks of running this:

• Both pass, similar diffs → pick on scope violations and diff readability. Smaller, in-scope diffs win; future-you doing code review will agree.
• One passes, one fails → rerun the failing leg once with an identical prompt before believing it. Variance is real; a single failure is weak evidence.
• Both fail → the task is probably underspecified. Fix TASK.md, not the models. This outcome has improved my prompt-writing more than any guide.
• Giant passing diff vs. small passing diff → I take the small one even if the large one looks cleverer. Revert cost dominates.

Limitations, and who should skip this

• This measures outcomes, not interaction quality. A model that passes tests but required you to watch it flail for ten minutes is worse than the table shows. If you care about the process, log the transcripts too.
• Timeouts distort results on large tasks. The 30-minute cap above is tuned for small, well-scoped tasks; for anything bigger, raise it and accept that slower legs are now systematically disadvantaged — note it in your results.
• Free tiers change. I am not claiming any specific quota, model lineup, or that free access lasts forever; check what is actually available when you run this, and design your experiment so it still means something if you have to swap one model out mid-matrix (keep pairings fixed within a task, at minimum).
• If your repo's tests are flaky, fix that first. This harness amplifies flakiness into garbage conclusions.
• If you only ever use one model and have no intention of switching, skip the bake-off entirely — your time is better spent writing better TASK.md files for the model you have.

The broader point: "which model is better" is a question with a per-repo, per-task answer, and the only way to get your answer is to run both against your code under controlled conditions. A worktree, a baseline gate, and a diff summary get you most of the way there for the cost of a shell script.