Here is the thing we want. You hand a coding agent a task, and before it starts you already know three things: what “done” means, how much it is allowed to spend getting there, and what it cannot do no matter what it is told.
None of that is written in capital letters in a prompt. I tried that. It is declared, a machine checks it, and the agent cannot argue with it.
That is a harness. In this part we build one against a small Go service with problems planted on purpose, run a real agent inside it, and watch four things happen: a sensor catches the agent’s confident first attempt and steers it right, a check on the tests themselves makes the agent write tests that would notice a broken line, a budget stops a run cold, and a request for safety that cannot be honoured refuses to pretend.
Tell a machine what good looks like
A rule in a prompt is advice. A sensor is a fact: a command whose exit code says whether the work is acceptable. The harness declares its sensors in a manifest, so anything that wants to know what “good” means for this repository can ask, without reading a word of prose.
Two sensors to start with, formatting and lint, and one verdict:
✗ gofmt 1 new
✗ lint 11 new
internal/clock/clock.go:11:14: Error return value ofos.WriteFile is not checked (errcheck)exit 1What this makes possible. The person at the terminal, the CI job and the agent in a loop stop disagreeing about what done means, because none of them decides it. Swap the vendor CLI underneath and the sensors stay put.
Forgive the past, refuse the new
Eleven lint findings are inherited debt. A gate that is red for ever teaches nobody anything, and one that is switched off teaches worse. So the ratchet records what is wrong today and fails only on what is new. Add one unchecked error, and that is all you see:
✗ lint 1 new, 11 known
internal/audit/audit.go:5:33: Error return value ofos.Remove is not checked (errcheck)What this makes possible. A gate on a ten-year-old repository today, without a week of cleanup first, and a standard that only goes one way from then on. Recording a baseline is a human act. The tool refuses it inside an agent run, so the thing being measured can never lower the bar it is measured against.
A definition of done the agent cannot see
Now the agent. It gets a task in plain words: fix the lint findings in internal/store, change nothing else.
The definition of done is mine. I wrote it as three small tests, and the harness declares them as a sensor. Then I kept them out of the repository, where the agent cannot read them. A test the agent can read is a test it plans around. I know, because the first time I tried this the test sat in the package, the agent read it while planning, and passed first time. A perfect score, and a useless demonstration.
An agent judged by acceptance it cannot see is in the same position as a student who was not handed the answers. One of the three:
func TestSaveIsPrivate(t *testing.T)// Saved files hold private data: owner read and write only.if fi.Mode().Perm() != 0o600 { t.Fatalf(“mode %v, want 0600”, …) }The agent ran with permission to edit files and nothing else, granted by me on the command line for that one run. It could not run the tests itself. Only the sensors could tell it whether it was done.
┌─ BUDGETS: turns, tokens, wall clock. The ceiling. ───────────┐
│ │
│ ┌────────────┐ edits ┌───────────────┐ │
│ │ AGENT │ ────────▶ │ WORKING TREE │ │
│ │ can edit, │ └───────┬───────┘ │
│ │ cannot run │ │ sensors check it │
│ └─────▲──────┘ ▼ │
│ │ ┌───────────────┐ │
│ └◀─ red: feedback ─┤ SENSORS (ynh) │ │
│ another turn │ run commands │ │
│ └───┬───────┬───┘ │
│ green: │ │ │
│ converged ◀─────┘ │ │
│ │ │
└───────────────────────────────────────┼──────────────────────┘
│ test-store only
▼
╔══════════════════════════════════╗
║ HIDDEN CONTRACT: outside the ║
║ agent's tree, never in the diff ║
╚══════════════════════════════════╝The agent can edit and cannot run anything. The contract sits behind a wall it never sees.
Here is what the sensors said, turn by turn:
turn 1 gofmt pass · lint known · lint-store pass · test-store fail
turn 2 gofmt pass · lint known · lint-store pass · test-store pass
exit 0, converged in 2 turnsTurn 1 fixed every lint finding and satisfied a stricter lint rule as well. It also still created the file readable by everyone. The contract failed, the failure went back as feedback, and turn 2 fixed it. In our run, the agent’s own explanation was that “the test failed because os.Create gives new files 0666 minus the umask”. It worked that out from the failure message alone.
That is the whole point of a sensor. Without it, the agent would have stopped at turn 1 with a confident summary, and I would have believed it, at least until somebody else’s process read the file.
It also changed a function signature nobody asked it to change. The sensors proved the contract holds. They did not prove the change is wise. A human still reviews, and a human still merges.
What this makes possible. Acceptance written before the work starts, by whoever owns quality, kept where the worker cannot game it. The same contract judges any agent and any model you swap in.
Tests that carry signal
A sensor that runs the tests tells you whether they pass. It does not tell you whether passing means anything. Tests and code that share the same wrong assumption pass together and carry no signal. Emmanuel Goossaert calls it the oracle problem in What I learned building an AI software factory, and his answer, a gate that checks the tests as well as the code, is the one I took.
So the harness gets three more checks, and they only run once everything else is green, because they are the expensive ones:
- Tests kept. Every test that passed before the change still runs and passes. An agent cannot delete a test, skip it, or add a
TestMainthat exits before anything runs. - Tests red. A new test has to fail against the old code. If it passes either way, it cannot tell the change apart from no change.
- Tests strong. Mutation testing on the lines the change touched: break each line on purpose, and some test has to notice.
turn 1 test-store fail turn 2 test-store pass · tests-kept pass · tests-red pass · tests-strong fail LIVED CONDITIONALS_NEGATION at internal/store/file.go:18:44 turn 3 test-store pass · tests-kept pass · tests-red pass · tests-strong pass exit 0, converged in 3 turns
On turn 2 the agent had fixed the store and passed my contract, and its new error handling in Remove could be inverted without any test noticing. On turn 3 it wrote two tests that do notice. Nobody asked it for tests. The check did.
The first time I ran this it got stuck instead. The fix changed Remove to return its error, so the agent’s new tests could not compile against the old code, and my “tests red” check counted that as a test that proves nothing. Nothing the agent could do would satisfy it, and five turns later ynh’s stuck detection ended the run. The check was wrong for this task, not the agent. “Tests red” fits fixes that keep the API, so this harness lets a change of API through.
What this makes possible. A green gate that means the tests were worth passing. The same three checks catch an agent that games its tests and an agent that writes weak ones, and the one that wrote weak ones is told which line its tests would not notice breaking.
A budget is a promise
Same task, one turn allowed:
exit 10 turn cap reached (1/1)
✗ test-store 2 newThe run stopped because it ran out of turns, not because it was finished, and the exit code says so.
That was not true a week ago. Writing this tutorial caught the loop sending the last feedback before checking the cap, so a one-turn run quietly did a second turn of work before it exited. It is fixed:
BEFORE: it leaked NOW: it holds
┌─────────────────────────────┐ ┌─────────────────────────────┐
│ turn 1 runs │ │ turn 1 runs │
├─────────────────────────────┤ ├─────────────────────────────┤
│ sensors say red │ │ sensors say red │
├─────────────────────────────┤ ├─────────────────────────────┤
│ feedback sent to agent │ │ cap checked first (1/1) │
├─────────────────────────────┤ ├─────────────────────────────┤
│ cap reached (1/1) │ │ stop, exit 10 │
├─────────────────────────────┤ ├─────────────────────────────┤
│ worker acts on feedback │ │ no feedback sent │
└─────────────────────────────┘ └─────────────────────────────┘
tree: a second turn of work tree: as turn 1 left itSame cap, one line moved. Check the ceiling before you hand over the feedback.
A budget that leaks a little is not a budget.
What this makes possible. The most a task can cost is known before it starts, which is what will let a factory run this same harness unattended in Part 4 without being handed a blank cheque.
Safety it cannot fake
The harness does not make a run safe and does not pretend to. Containment is chosen by the operator, per run. The interesting question is what happens when you ask for containment that isn’t there:
exit 20 starting worker: srt not found on PATH
turns 0 · tokens 0 · 11 msHere is every path through that decision:
Was a sandbox asked for? (--sandbox)
│
├─ none ─────────────▶ RUNS, deliberately unsandboxed
│
└─ srt
│
Is srt installed?
├─ no ───────────▶ REFUSES: exit 20, before any model call
│
└─ yes
│
Does the backend support it?
├─ codex ────▶ REFUSES: exit 1, with the reason
│
└─ claude ───▶ RUNS, inside srtEvery path ends in a run or a refusal. None ends in a quiet fallback.
A safety control that silently does not apply is worse than none, because people rely on it.
What this makes possible. “This lane always runs sandboxed” becomes a sentence you can write down and know is true, because the alternative is a refusal, not a surprise.
The Debrief
TL;DR
A harness is everything around the model that says what “done” means, how much a run may spend, and what the agent cannot do. This part builds one for a small Go service with planted problems and runs a real agent inside it.
The agent’s first attempt fixed every lint finding and still left saved files readable by everyone. A contract it could not see failed, the failure went back as feedback, and the second turn fixed it. Then three checks on the tests themselves found code its tests would not notice breaking, and a third turn added the tests. With one turn allowed, the run stopped with exit 10. Asked for a sandbox that was not installed, it refused to run at all.
The ideas
- Sensors are checks declared as commands with exit codes, so a person, CI and an agent all read the same verdict.
- A ratchet fails only on new problems, so a gate can go on a legacy repository today.
- The definition of done lives outside the repository, where the agent cannot plan around it.
- Passing tests only count if the tests carry signal: none lost, new ones that fail on the old code, and no surviving mutants on the changed lines.
- The budget is checked before feedback is sent, so a capped run cannot sneak in another turn.
- Missing containment means a refusal, never a quiet fallback.
What to take away
- Turn your definition of done into commands with exit codes before you hand an agent a task.
- Keep acceptance tests where the agent cannot read them, and check the tests as well as the code.
- Check that your budgets and sandboxes actually hold. Ours leaked, at first.
Tutorial
You will build the whole harness from an empty folder by copy and paste, run a real agent inside it three times, and watch the refusal. About forty minutes. Three model runs, well under two dollars: in our runs $0.40, $0.32 and $0.53.
1. Install the tools
brew install eyelock/tap/ynh eyelock/tap/ynm eyelock/tap/ynf
ynh --version # tested with 0.12.1
ynm --version # tested with 0.4.1
ynf --version # tested with 0.2.0On Linux, the same versions are on each project’s GitHub releases page: ynh, ynm and ynf.
You also need Go, golangci-lint v2, jq, git with an identity set (user.name and user.email) and Claude Code signed in. Step 8 also needs gremlins, a mutation tester for Go, on your PATH: go install github.com/go-gremlins/gremlins/cmd/gremlins@v0.6.0, which puts it in $(go env GOPATH)/bin. Docker is optional, for the last section only. The commands are written for bash: on macOS, run bash first.
2. Start clean
Everything lives under one folder, and the tools get their own homes there, so nothing touches your own setup. Run this every time you start or restart the part:
export LTL="$HOME/ltl" # any folder you like; keep the path short
export YNH_HOME="$LTL/.home/ynh"
export GOLANGCI_LINT_CACHE="$LTL/.home/golangci-lint"
mkdir -p "$YNH_HOME" "$GOLANGCI_LINT_CACHE"Now remove anything a previous run of this part left behind:
ynh uninstall local/leash 2>/dev/null
docker rmi leash:part1 ltl-ynh-go 2>/dev/null
rm -rf "$LTL/part1" && mkdir -p "$LTL/part1" && cd "$LTL/part1"Then fetch the target service fresh. It is a small Go service with problems planted on purpose, and it is public, in the ynf repository:
curl -sL https://github.com/eyelock/ynf/archive/refs/tags/v0.2.0.tar.gz | tar xz
cp -R ynf-0.2.0/sandbox/seed work && rm -rf ynf-0.2.0 work/.agents
cd work && git init -q && git add -A && git commit -qm "target as received"The folder $LTL/part1 now holds work/, the repository the agent will tidy. The harness you build lives next to it in $LTL/part1/harness/, and the reason it lives outside the repository matters in step 5. Every command below runs from work/ unless it says otherwise.
Coming back later: re-run the exports above, then cd "$LTL/part1/work". The installed harness is remembered in $YNH_HOME, so nothing else needs redoing.
3. Tell a machine what good looks like
Create the first version of the harness: a manifest with two sensors, formatting and lint, and a line of project instructions.
mkdir -p "$LTL/part1/harness/.agents/harness"
cat > "$LTL/part1/harness/.agents/harness/plugin.json" <<'EOF'
{
"$schema": "https://eyelock.github.io/ynh/schema/plugin.schema.json",
"name": "leash",
"version": "0.1.0",
"default_vendor": "claude",
"sensors": {
"gofmt": {
"category": "maintainability",
"tolerance": "blocking",
"version_command": "go version",
"source": {
"command": "test -z \"$(gofmt -l .)\" || { gofmt -l . ; exit 1; }"
},
"output": {
"format": "text"
}
},
"lint": {
"category": "maintainability",
"tolerance": "blocking",
"version_command": "golangci-lint version",
"source": {
"command": "GOLANGCI_LINT_CACHE=\"$PWD/.cache/golangci-lint\" golangci-lint run ./..."
},
"output": {
"format": "text",
"match": "^[^ ]+\\.go:[0-9]+:[0-9]+:"
}
}
}
}
EOF
cat > "$LTL/part1/harness/AGENTS.md" <<'EOF'
# leash
Project-level instructions that apply to every session with this harness.
EOF
cd "$LTL/part1/work"
ynd validate ../harness
ynh install ../harness
ynh sensors ls local/leashYou should see ../harness: valid followed by a list of what it checked, then Installed harness "leash", then the two sensors:
NAME CATEGORY SOURCE FORMAT
gofmt maintainability command text
lint maintainability command textNothing has run yet. Now ask for a verdict:
ynh check local/leash ; echo "exit $?"You should see, trimmed, and exit 1:
✗ gofmt 1 new 15ms
✗ lint 11 new 939ms
gofmt:
internal/format/format.go
lint:
internal/clock/clock.go:11:14: Error return value of `os.WriteFile` is not checked (errcheck)
...
exit 1Fix the formatting and ask again:
gofmt -w internal/format/format.go
ynh check local/leash ; echo "exit $?"
git commit -qam "gofmt format.go"gofmt turns green, lint stays red with eleven findings, exit 1. If you see exit 2, a sensor could not run at all: check golangci-lint is on your PATH.
4. Forgive the past, refuse the new
Record what is wrong today as a baseline, then check again:
ynh check local/leash --update-baseline
ynh check local/leash ; echo "exit $?"
git add -A && git commit -qm "record baseline"You should see ~ lint known (11) and exit 0. The baseline is one small file of fingerprints in .ynh/baseline/, committed with the code. Now add one new problem:
mkdir -p internal/audit
printf 'package audit\n\nimport "os"\n\nfunc Purge(p string) { os.Remove(p) }\n' > internal/audit/audit.go
ynh check local/leash ; echo "exit $?"You should see exit 1, and only the newcomer:
✓ gofmt pass 17ms
✗ lint 1 new, 11 known 297ms
lint - 1 new (11 pre-existing not shown):
internal/audit/audit.go:5:33: Error return value of `os.Remove` is not checked (errcheck)Take it out again so the next step starts clean:
rm -r internal/audit && ynh check local/leash ; echo "exit $?"Exit 0 again. If you see more than one new finding, the baseline commit above did not happen.
5. A definition of done the agent cannot see
Write the second version of the harness over the first. It adds a hidden contract (three small tests), a stricter lint rule for internal/store, a script that runs the contract, two task-scoped sensors and a budget. Start with the contract and the two helper files:
▸ Show the contract and helper files (71 lines, open and copy the whole block)
mkdir -p ../harness/contract ../harness/tools
cat > ../harness/contract/store_contract_test.go.txt <<'EOF'
package store_test
import (
"os"
"path/filepath"
"testing"
"github.com/eyelock/ynf-sandbox/internal/store"
)
// Saved files hold private data: owner read and write only.
func TestSaveIsPrivate(t *testing.T) {
p := filepath.Join(t.TempDir(), "blob")
if err := store.Save(p, []byte("x")); err != nil {
t.Fatal(err)
}
fi, err := os.Stat(p)
if err != nil {
t.Fatal(err)
}
if fi.Mode().Perm() != 0o600 {
t.Fatalf("mode %v, want 0600", fi.Mode().Perm())
}
}
// Saving over an existing, longer file leaves exactly the new data.
func TestSaveReplacesWholly(t *testing.T) {
p := filepath.Join(t.TempDir(), "blob")
if err := store.Save(p, []byte("a long first version")); err != nil {
t.Fatal(err)
}
if err := store.Save(p, []byte("short")); err != nil {
t.Fatal(err)
}
b, err := os.ReadFile(p)
if err != nil || string(b) != "short" {
t.Fatalf("got %q, %v", b, err)
}
}
// Saving into a directory that does not exist is an error, not a silent success.
func TestSaveMissingDirFails(t *testing.T) {
p := filepath.Join(t.TempDir(), "absent", "blob")
if err := store.Save(p, []byte("x")); err == nil {
t.Fatal("expected an error")
}
}
EOF
cat > ../harness/tools/test-store.sh <<'EOF'
#!/bin/sh
set -e
overlay=$(mktemp)
trap 'rm -f "$overlay"' EXIT
printf '{"Replace":{"%s/internal/store/zz_contract_test.go":"%s/contract/store_contract_test.go.txt"}}' \
"$PWD" "$YNH_HARNESS_DIR" > "$overlay"
go test -overlay "$overlay" ./internal/store/
EOF
chmod +x ../harness/tools/test-store.sh
cat > ../harness/golangci-store.yml <<'EOF'
version: "2"
run:
relative-path-mode: wd
linters:
default: standard
settings:
errcheck:
check-blank: true
EOFThe test-store script maps the contract into the package for one go test process with -overlay, so it never appears in the agent’s working tree, the diff or the commit. Now the manifest that declares it all, replacing the first one:
cat > ../harness/.agents/harness/plugin.json <<'EOF'
{
"$schema": "https://eyelock.github.io/ynh/schema/plugin.schema.json",
"name": "leash",
"version": "0.1.0",
"default_vendor": "claude",
"agent": {
"max_turns": 12,
"max_tokens": 600000,
"max_wall": "10m"
},
"sensors": {
"gofmt": {
"category": "maintainability",
"tolerance": "blocking",
"version_command": "go version",
"source": {
"command": "test -z \"$(gofmt -l .)\" || { gofmt -l . ; exit 1; }"
},
"output": {
"format": "text"
}
},
"lint": {
"category": "maintainability",
"tolerance": "blocking",
"version_command": "golangci-lint version",
"source": {
"command": "GOLANGCI_LINT_CACHE=\"$PWD/.cache/golangci-lint\" golangci-lint run ./..."
},
"output": {
"format": "text",
"match": "^[^ ]+\\.go:[0-9]+:[0-9]+:"
}
},
"lint-store": {
"category": "maintainability",
"tolerance": "blocking",
"version_command": "golangci-lint version",
"source": {
"command": "GOLANGCI_LINT_CACHE=\"$PWD/.cache/golangci-lint\" golangci-lint run -c \"$YNH_HARNESS_DIR/golangci-store.yml\" ./internal/store/..."
},
"output": {
"format": "text",
"match": "^[^ ]+\\.go:[0-9]+:[0-9]+:"
}
},
"test-store": {
"category": "behaviour",
"tolerance": "blocking",
"version_command": "go version",
"source": {
"command": "\"$YNH_HARNESS_DIR/tools/test-store.sh\""
},
"output": {
"format": "text",
"match": "^(--- FAIL|[^ ]+\\.go:[0-9]+)"
}
}
}
}
EOFGive the task a name, reinstall, accept the trust prompt and look at the starting position:
ynh focus add ../harness tidy "Fix the golangci-lint findings in internal/store, and change nothing outside internal/store. Stop when the change is ready: do not commit."
ynd validate ../harness && ynh uninstall local/leash && ynh install ../harness
ynh trust accept local/leash
ynh check local/leash ; echo "exit $?"ynh trust accept records that you have reviewed the commands this harness will run. Sensors are commands on your machine, so ynh will not run a new or changed harness until you say so. You should see Added focus "tidy", Recorded local/leash: 8 command(s), and the starting position, red on both scoped sensors, exit 1:
✓ gofmt pass 18ms
~ lint known (11) 274ms
✗ lint-store 3 new 796ms
✗ test-store 2 new 673msNow run the agent. This is model run A:
ynh agent run --harness local/leash --focus tidy --auto-approve edits --max-turns 8 \
--format json --emit-jsonl ../trajectory.jsonl > ../run.json ; echo "exit $?"--auto-approve edits is you, the operator, granting the agent permission to edit files for this one run. The harness cannot grant it, and it does not include running commands, so only the sensors can tell the agent whether it is done. A little under two minutes later you should see exit 0. Read why it took two turns:
jq -r 'select(.type=="sensor_result") | "turn \(.turn) \(.data.name) \(.data.status)"' ../trajectory.jsonlturn 1 gofmt pass
turn 1 lint known
turn 1 lint-store pass
turn 1 test-store fail
turn 2 gofmt pass
turn 2 lint known
turn 2 lint-store pass
turn 2 test-store passTurn 1 fixed every lint finding and still created the file readable by everyone; the failure went back as feedback and turn 2 fixed it. Your run will differ in its words and token count, and if it converges in one turn it happened to get the permissions right first time, which is rare. Check what it touched:
git diff --stat
ynh check local/leash ; echo "exit $?"You should see two files, both in internal/store, and a green gate, exit 0:
✓ gofmt pass 39ms
~ lint known (8) 437ms
✓ lint-store pass 235ms
✓ test-store pass 99ms
ok: 3 passed, 1 knownThe baseline: 3 recorded failures are now fixed line below it is the ratchet noticing the improvement. Read git diff before you believe the change: the sensors proved the contract holds, not that every edit is wise.
6. A budget is a promise
Put the tree back and give the same task one turn. This is model run B:
git checkout -q -- . && git clean -fdq
ynh agent run --harness local/leash --focus tidy --auto-approve edits --max-turns 1 \
--format json --emit-jsonl ../trajectory-1.jsonl > ../run-1.json ; echo "exit $?"
ynh check local/leash ; echo "exit $?"You should see exit 10 from the run, then a red gate, exit 1 (trimmed):
✓ gofmt pass 18ms
~ lint known (8) 300ms
✓ lint-store pass 231ms
✗ test-store 2 new 225ms
blocked: 1 of 4 sensors failedThe run stopped because it ran out of turns, and the exit code says so. If you see exit 0 instead, the agent got it right in a single turn; run it once more. Reset before the next step:
git checkout -q -- . && git clean -fdq7. Safety it cannot fake
Ask for a sandbox that is not there. This makes no model call:
ynh agent run --harness local/leash --focus tidy --sandbox srt --max-turns 1 --format json \
| jq '{exit_code, reason, consumed}'Unless you have srt (Anthropic’s sandbox runtime) installed, you should see exit 20 in a few milliseconds:
{
"exit_code": 20,
"reason": "starting worker: srt not found on PATH: exec: \"srt\": executable file not found in $PATH",
"consumed": { "turns": 0, "tokens": 0, "wall_ms": 11 }
}It did not shrug and run unsandboxed. Ask for a sandbox the backend cannot provide and it refuses in the same way:
ynh agent run --harness local/leash --focus tidy --sandbox srt --backend codex --max-turns 1Error: --sandbox srt is not supported by the codex backend (only claude implements it); run inside a container you configured, or use --sandbox none to proceed deliberately unsandboxedOptional: the same harness, in a box
With Docker, the harness can be baked into an image and its gate run with no network at all. The base image carries ynh and the vendor CLIs; this small Dockerfile adds Go and golangci-lint for these sensors.
mkdir -p "$LTL/part1/image"
cat > "$LTL/part1/image/Dockerfile" <<'EOF'
# Harness base with the Go toolchain and golangci-lint, on top of the public ynh image.
FROM golang:1.27-alpine AS go
FROM golangci/golangci-lint:v2.14.0-alpine AS lint
FROM ghcr.io/eyelock/ynh:0.12.0
COPY --from=go /usr/local/go /usr/local/go
COPY --from=lint /usr/bin/golangci-lint /usr/local/bin/golangci-lint
ENV PATH=/usr/local/go/bin:$PATH
EOF
docker build -q -t ltl-ynh-go "$LTL/part1/image"
ynh image local/leash --tag leash:part1 --base ltl-ynh-go
docker run --rm --network none -v "$PWD":/workspace -w /workspace --entrypoint ynh \
leash:part1 check local/leash ; echo "exit $?"You should see Harness image built: leash:part1, then the same red gate you saw at the start of step 5, exit 1, with no network. Running the agent itself in there needs model credentials passed in for the one run, never baked into the image.
8. Check the tests themselves
Everything so far asks whether the tests pass. This step asks whether passing means anything: the oracle problem, in Emmanuel Goossaert’s phrase from What I learned building an AI software factory. Three checks on the tests go into the harness as convergence verifiers, so they run only once every other sensor is green, cheapest first.
The checker is one small Go program and a wrapper that builds it fresh for each run. It lives in the harness, out of the agent’s reach, because a check the agent can edit checks nothing:
▸ Show the checker (go.mod, main.go, run.sh; 537 lines, open and copy the whole block)
mkdir -p ../harness/test-trust/testtrust
cat > ../harness/test-trust/testtrust/go.mod <<'EOF'
module testtrust
go 1.24
EOF
cat > ../harness/test-trust/testtrust/main.go <<'EOF'
// Command testtrust asks whether a Go repository's tests still carry signal after a change.
//
// testtrust set Every test that passed at the base still runs and passes.
// testtrust red Every newly named test fails against the base code, three times out of three.
// Only new test names are checked: a strengthened assertion in an existing
// test is not.
// testtrust strong Every mutant gremlins makes on the changed lines is killed by a test.
// Findings are gremlins' own lines: "LIVED <MUTATOR> at <file>:<line>:<col>"
// or "NOT COVERED ...". Needs gremlins on PATH and the whole suite green.
//
// The base is $TEST_TRUST_BASE, the commit the run started from, which whatever drives the run
// must set. Without it, the merge base with origin/HEAD is used, and without that the check
// refuses to run: defaulting to HEAD would compare the change with itself once anything commits.
// The change is the working tree, untracked files included. The repository root must be the Go
// module root.
//
// It is meant to run as a ynh sensor with the convergence-verifier role, from a harness the agent
// cannot write to. Each finding is one line, "KEPT <package> <test>: <what>" or "RED ...", so
// output.match can be ^(KEPT|RED) and no narrative or error line can ever match it.
// Exit 0 is a pass and 1 is a finding. Exit 2 means the check itself could not run; its message
// is for the operator, because there is nothing the agent can change to fix it.
//
// Settings: TEST_TRUST_TIMEOUT (each go test, default 5m), TEST_TRUST_DEADLINE (the whole check,
// default 20m), TEST_TRUST_MAX_MUTANTS (strong refuses a diff with more, default 50),
// TEST_TRUST_TIMEOUT_COEFFICIENT (gremlins' per-mutant time limit, default 50),
// TEST_TRUST_REQUIRE_TESTS=1 (a change with no new tests is a finding),
// TEST_TRUST_ALLOW_NEW_API=1 (new tests that only fail to build at the base, because they name
// something new, are not a finding; for feature lanes).
package main
import (
"bufio"
"bytes"
"context"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"os"
"os/exec"
"path/filepath"
"regexp"
"sort"
"strconv"
"strings"
"time"
)
var (
ctx = context.Background()
timeout string
temps []string
)
func main() {
if len(os.Args) != 2 || (os.Args[1] != "set" && os.Args[1] != "red" && os.Args[1] != "strong") {
fmt.Fprintln(os.Stderr, "usage: testtrust set|red|strong")
os.Exit(2)
}
deadline, err := time.ParseDuration(envOr("TEST_TRUST_DEADLINE", "20m"))
if err != nil {
fail("TEST_TRUST_DEADLINE: %v", err)
}
var cancel context.CancelFunc
ctx, cancel = context.WithTimeout(context.Background(), deadline)
defer cancel()
timeout = envOr("TEST_TRUST_TIMEOUT", "5m")
top, err := output(".", "git", "rev-parse", "--show-toplevel")
if err != nil {
fail("not inside a git repository: %v", err)
}
if err := os.Chdir(strings.TrimSpace(top)); err != nil {
fail("%v", err)
}
if _, err := os.Stat("go.mod"); err != nil {
fail("no go.mod at the repository root; a module in a subdirectory is not supported")
}
sha := resolveBase()
var findings []string
switch os.Args[1] {
case "set":
findings = testSet(sha)
case "red":
findings = redFirst(sha)
case "strong":
findings = strong(sha)
}
cleanup()
prefix := map[string]string{"set": "KEPT ", "red": "RED ", "strong": ""}[os.Args[1]]
for _, f := range findings {
fmt.Println(prefix + f)
}
if len(findings) > 0 {
os.Exit(1)
}
}
func resolveBase() string {
base := os.Getenv("TEST_TRUST_BASE")
if base == "" {
mb, err := output(".", "git", "merge-base", "HEAD", "origin/HEAD")
if err != nil {
fail("TEST_TRUST_BASE is not set and there is no origin/HEAD to fall back to; set it to the commit the run started from")
}
base = strings.TrimSpace(mb)
}
sha, err := output(".", "git", "rev-parse", "--verify", base+"^{commit}")
if err != nil {
fail("cannot resolve TEST_TRUST_BASE %q to a commit", base)
}
return strings.TrimSpace(sha)
}
// testSet fails on any test that passed at the base and is now missing, skipped, failing or not
// building. It catches tests that were deleted or skipped, and new files that stop the old tests
// running (a TestMain that exits early, an init that calls os.Exit).
func testSet(sha string) []string {
before := cachedBase(sha)
after, buildFailed := run(".", "./...")
var findings []string
for _, k := range sortedKeys(before) {
if before[k] != "pass" {
continue
}
switch got, ok := after[k]; {
case !ok && buildFailed[pkgOf(k)]:
findings = append(findings, k+": does not build (passed at base)")
case !ok:
findings = append(findings, k+": no longer runs (passed at base)")
case got != "pass":
findings = append(findings, k+": now "+got+" (passed at base)")
}
}
if len(findings) == 0 {
fmt.Fprintf(os.Stderr, "tests kept: all %d tests that passed at %s still pass\n", countPass(before), sha[:7])
}
return findings
}
// redFirst fails on any newly named test that passes against the base code: a test that cannot
// tell the change apart from no change. New tests must fail as tests, not as a build, so tests
// that only fail because they name something new are a finding too, unless the lane allows it.
func redFirst(sha string) []string {
files := changedTests(sha)
if len(files) == 0 {
fmt.Fprintln(os.Stderr, "tests red: no new or changed test files")
if os.Getenv("TEST_TRUST_REQUIRE_TESTS") == "1" {
return []string{"(change) tests: adds no tests"}
}
return nil
}
tree := extract(sha)
newTests := map[string][]string{} // package dir -> new test names
for _, f := range files {
dir := "./" + filepath.Dir(f)
if _, done := newTests[dir]; done {
continue
}
after, err := list(".", dir)
if err != nil {
fail("go test -list %s fails on the change itself, so its tests cannot be compared: %v", dir, err)
}
was := map[string]bool{}
if before, err := list(tree, dir); err == nil {
for _, n := range before {
was[n] = true
}
} // a package that does not list at the base is new there: every test in it is new
newTests[dir] = []string{}
for _, n := range after {
if !was[n] {
newTests[dir] = append(newTests[dir], n)
}
}
}
for _, f := range files {
b, err := os.ReadFile(f)
if err != nil {
fail("reading %s: %v", f, err)
}
dst := filepath.Join(tree, f)
if err := os.MkdirAll(filepath.Dir(dst), 0o755); err != nil {
fail("%v", err)
}
if err := os.WriteFile(dst, b, 0o644); err != nil {
fail("%v", err)
}
}
var findings []string
total := 0
for _, dir := range sortedKeys(newTests) {
names := newTests[dir]
if len(names) == 0 {
continue
}
total += len(names)
res, buildFailed := run(tree, dir, "-count=3", "-run", "^("+strings.Join(names, "|")+")$")
pkg := pkgPath(".", dir)
if buildFailed[pkg] {
if os.Getenv("TEST_TRUST_ALLOW_NEW_API") == "1" {
fmt.Fprintf(os.Stderr, "tests red: %s new tests name something new, allowed by TEST_TRUST_ALLOW_NEW_API\n", pkg)
continue
}
findings = append(findings, pkg+" (new tests): do not build against the base code, so they cannot show they detect the change")
continue
}
for _, n := range names {
k := pkg + " " + n
switch res[k] {
case "fail":
case "pass":
findings = append(findings, k+": passes against the base code, so it cannot tell the change apart")
case "skip":
findings = append(findings, k+": skips against the base code, so it shows nothing")
default:
findings = append(findings, k+": did not run against the base code")
}
}
}
if len(findings) == 0 && total > 0 {
fmt.Fprintf(os.Stderr, "tests red: %d new test(s) fail against %s, three times out of three\n", total, sha[:7])
}
return findings
}
var (
runnable = regexp.MustCompile(`(?m)^Runnable: (\d+)`)
survivor = regexp.MustCompile(`(?m)^ *((?:LIVED|NOT COVERED) .*)$`)
timedOut = regexp.MustCompile(`(?m)^Timed out: (\d+)`)
)
// strong runs gremlins on the lines changed since the base. A dry run first counts the mutants,
// so a diff too big to test in time is refused up front instead of overrunning the budget.
func strong(sha string) []string {
if _, err := exec.LookPath("gremlins"); err != nil {
fail("gremlins is not on PATH; go install github.com/go-gremlins/gremlins/cmd/gremlins@v0.6.0")
}
// With no changed code, gremlins' --diff selects nothing and it falls back to the whole
// module, so an untouched tree would be judged on every line it has.
if len(changedCode(sha)) == 0 {
fmt.Fprintln(os.Stderr, "tests strong: no changed Go code to mutate")
return nil
}
max, err := strconv.Atoi(envOr("TEST_TRUST_MAX_MUTANTS", "50"))
if err != nil {
fail("TEST_TRUST_MAX_MUTANTS: %v", err)
}
dry, err := output(".", "gremlins", "unleash", "--dry-run", "--diff", sha, ".")
if err != nil {
fail("gremlins dry run failed, usually because the test suite is not green: %v\n%s", err, tail(dry))
}
m := runnable.FindStringSubmatch(dry)
if m == nil {
fail("gremlins dry run printed no mutant count:\n%s", tail(dry))
}
if n, _ := strconv.Atoi(m[1]); n > max {
fail("the change has %d mutants on its lines, over TEST_TRUST_MAX_MUTANTS (%d); raise it or split the change", n, max)
}
// gremlins times each mutant against the coverage run, which on a small, fast suite is shorter
// than compiling the mutant: a low coefficient times mutants out instead of testing them.
out, err := output(".", "gremlins", "unleash", "--timeout-coefficient", envOr("TEST_TRUST_TIMEOUT_COEFFICIENT", "50"), "--diff", sha, ".")
if errors.Is(ctx.Err(), context.DeadlineExceeded) {
fail("ran past TEST_TRUST_DEADLINE")
}
if err != nil {
fail("gremlins failed: %v\n%s", err, tail(out))
}
if t := timedOut.FindStringSubmatch(out); t != nil && t[1] == m[1] && t[1] != "0" {
fail("all %s mutants timed out, so none was tested; raise TEST_TRUST_TIMEOUT_COEFFICIENT", t[1])
}
var findings []string
for _, f := range survivor.FindAllStringSubmatch(out, -1) {
findings = append(findings, strings.TrimSpace(f[1]))
}
if len(findings) == 0 {
fmt.Fprintf(os.Stderr, "tests strong: every mutant on the changed lines was killed (%s runnable)\n", m[1])
}
return findings
}
func tail(s string) string {
lines := strings.Split(strings.TrimSpace(s), "\n")
if len(lines) > 15 {
lines = lines[len(lines)-15:]
}
return strings.Join(lines, "\n")
}
// cachedBase runs the whole suite at the base once per commit and Go version, three times so that
// only tests that pass every time count as passing; it is the expensive half. The cache lives
// in the user's cache folder, outside the worktree, so the agent cannot forge it, and a result
// with no passing tests is refused rather than cached: it would make every change pass.
func cachedBase(sha string) map[string]string {
cache := cachePath(sha)
if b, err := os.ReadFile(cache); err == nil {
var m map[string]string
if json.Unmarshal(b, &m) == nil && countPass(m) > 0 {
return m
}
}
tree := extract(sha)
res, _ := run(tree, "./...", "-count=3")
if countPass(res) == 0 {
fail("the base %s ran no passing tests, so there is nothing to compare against; check that it builds and its modules download (git archive leaves out submodules and LFS files)", sha[:7])
}
if b, err := json.Marshal(res); err == nil && os.MkdirAll(filepath.Dir(cache), 0o755) == nil {
tmp := cache + fmt.Sprintf(".%d.tmp", os.Getpid())
if os.WriteFile(tmp, b, 0o644) == nil {
_ = os.Rename(tmp, cache)
}
}
return res
}
func cachePath(sha string) string {
dir, err := os.UserCacheDir()
if err != nil {
fail("no user cache folder: %v", err)
}
repo, err := output(".", "git", "config", "--get", "remote.origin.url")
if err != nil || strings.TrimSpace(repo) == "" {
repo, _ = os.Getwd()
}
key := sha256.Sum256([]byte(strings.TrimSpace(repo)))
goVersion, err := output(".", "go", "env", "GOVERSION")
if err != nil {
fail("go env GOVERSION: %v", err)
}
return filepath.Join(dir, "test-trust", hex.EncodeToString(key[:6]), sha+"-"+strings.TrimSpace(goVersion)+".json")
}
// run executes go test -json in dir and returns each test's final outcome, keyed
// "<package> <test>", and which packages failed to build. Any run that failed is a "fail",
// so a test that passes once in three is not counted as passing.
func run(dir string, args ...string) (map[string]string, map[string]bool) {
cmd := exec.CommandContext(ctx, "go", append([]string{"test", "-json", "-timeout", timeout}, args...)...)
cmd.Dir = dir
var out bytes.Buffer
cmd.Stdout = &out
cmd.Stderr = &out
_ = cmd.Run() // a failing test is an answer, not an error
if errors.Is(ctx.Err(), context.DeadlineExceeded) {
fail("ran past TEST_TRUST_DEADLINE")
}
res, built := map[string]string{}, map[string]bool{}
sc := bufio.NewScanner(&out)
sc.Buffer(make([]byte, 1<<20), 1<<24)
for sc.Scan() {
var e struct{ Action, Package, Test, ImportPath string }
if json.Unmarshal(sc.Bytes(), &e) != nil {
continue
}
switch {
case e.Action == "build-fail":
built[e.ImportPath] = true
case e.Test == "" && e.Action == "fail" && e.Package != "" && !hasTest(res, e.Package):
built[e.Package] = true
case e.Test != "" && (e.Action == "pass" || e.Action == "fail" || e.Action == "skip"):
k := e.Package + " " + e.Test
if res[k] != "fail" {
res[k] = e.Action
}
}
}
return res, built
}
func list(dir, pkg string) ([]string, error) {
cmd := exec.CommandContext(ctx, "go", "test", "-list", ".", pkg)
cmd.Dir = dir
out, err := cmd.Output()
if err != nil {
return nil, err
}
var names []string
for _, l := range strings.Split(string(out), "\n") {
if strings.HasPrefix(l, "Test") || strings.HasPrefix(l, "Fuzz") || strings.HasPrefix(l, "Example") {
names = append(names, strings.TrimSpace(l))
}
}
return names, nil
}
func changedTests(sha string) []string { return changed(sha, true) }
func changedCode(sha string) []string { return changed(sha, false) }
// changed lists Go files added or modified since the base, untracked ones included: test files
// when tests is true, the rest otherwise.
func changed(sha string, tests bool) []string {
diff, err := output(".", "git", "diff", "--name-only", "--diff-filter=AMR", sha, "--", "*.go")
if err != nil {
fail("git diff %s: %v", sha[:7], err)
}
untracked, err := output(".", "git", "ls-files", "--others", "--exclude-standard", "--", "*.go")
if err != nil {
fail("git ls-files: %v", err)
}
seen := map[string]bool{}
var files []string
for _, f := range strings.Fields(diff + "\n" + untracked) {
if strings.HasSuffix(f, "_test.go") != tests {
continue
}
if !seen[f] {
seen[f] = true
files = append(files, f)
}
}
sort.Strings(files)
return files
}
// extract writes the base commit's tree to a temporary folder, leaving the repository untouched.
func extract(sha string) string {
tmp, err := os.MkdirTemp("", "testtrust-")
if err != nil {
fail("%v", err)
}
temps = append(temps, tmp)
archive := exec.CommandContext(ctx, "git", "archive", sha)
untar := exec.CommandContext(ctx, "tar", "-x", "-C", tmp)
pipe, err := archive.StdoutPipe()
if err != nil {
fail("%v", err)
}
untar.Stdin = pipe
if err := untar.Start(); err != nil {
fail("tar: %v", err)
}
if err := archive.Run(); err != nil {
fail("git archive %s: %v", sha[:7], err)
}
if err := untar.Wait(); err != nil {
fail("tar: %v", err)
}
return tmp
}
func pkgPath(dir, rel string) string {
cmd := exec.CommandContext(ctx, "go", "list", rel)
cmd.Dir = dir
out, err := cmd.Output()
if err != nil {
return rel
}
return strings.TrimSpace(string(out))
}
func hasTest(res map[string]string, pkg string) bool {
for k := range res {
if strings.HasPrefix(k, pkg+" ") {
return true
}
}
return false
}
func pkgOf(k string) string { return strings.SplitN(k, " ", 2)[0] }
func countPass(m map[string]string) int {
n := 0
for _, v := range m {
if v == "pass" {
n++
}
}
return n
}
func sortedKeys[V any](m map[string]V) []string {
keys := make([]string, 0, len(m))
for k := range m {
keys = append(keys, k)
}
sort.Strings(keys)
return keys
}
func envOr(name, def string) string {
if v := os.Getenv(name); v != "" {
return v
}
return def
}
func output(dir, name string, args ...string) (string, error) {
cmd := exec.CommandContext(ctx, name, args...)
cmd.Dir = dir
out, err := cmd.Output()
return string(out), err
}
func cleanup() {
for _, t := range temps {
_ = os.RemoveAll(t)
}
temps = nil
}
// fail is for the operator: the check could not run, and nothing the agent changes will fix it.
func fail(format string, args ...any) {
cleanup()
fmt.Fprintf(os.Stderr, "testtrust: cannot check (operator, not the agent): "+format+"\n", args...)
os.Exit(2)
}
EOF
cat > ../harness/test-trust/run.sh <<'EOF'
#!/bin/sh
# Runs testtrust (set, red or strong), built fresh for this run into a temporary folder and
# removed after: there is no binary left at a known path to swap for one that always passes.
# The Go build cache keeps this to about a second. Building rather than `go run` keeps the exit
# code: 0 pass, 1 finding, 2 cannot check (go run turns 2 into 1). The target's GOFLAGS and
# workspace are kept out of this build, and left alone for the tests the checks run.
src=$(cd "$(dirname "$0")/testtrust" && pwd) || exit 2
dir=$(mktemp -d) || exit 2
GOFLAGS= GOWORK=off go -C "$src" build -o "$dir/testtrust" . || {
rm -rf "$dir"
echo "testtrust: cannot check (operator, not the agent): build failed" >&2
exit 2
}
"$dir/testtrust" "$@"
status=$?
rm -rf "$dir"
exit $status
EOF
chmod +x ../harness/test-trust/run.shNow declare the three checks, reinstall, and tell them which commit the work started from:
jq '.sensors += {
"tests-kept": {
"category": "behaviour", "role": "convergence-verifier", "tolerance": "blocking",
"version_command": "go version",
"source": { "command": "\"$YNH_HARNESS_DIR/test-trust/run.sh\" set" },
"output": { "format": "text", "match": "^(KEPT|RED) " }
},
"tests-red": {
"category": "behaviour", "role": "convergence-verifier", "tolerance": "blocking",
"version_command": "go version",
"source": { "command": "TEST_TRUST_ALLOW_NEW_API=1 \"$YNH_HARNESS_DIR/test-trust/run.sh\" red" },
"output": { "format": "text", "match": "^(KEPT|RED) " }
},
"tests-strong": {
"category": "behaviour", "role": "convergence-verifier", "tolerance": "blocking",
"version_command": "gremlins --version",
"source": { "command": "CI=1 \"$YNH_HARNESS_DIR/test-trust/run.sh\" strong" },
"output": { "format": "text", "match": "^(LIVED|NOT COVERED) " }
}
}' ../harness/.agents/harness/plugin.json > ../plugin.json && mv ../plugin.json ../harness/.agents/harness/plugin.json
ynd validate ../harness >/dev/null && ynh uninstall local/leash >/dev/null && ynh install ../harness >/dev/null
ynh trust accept local/leash
export TEST_TRUST_BASE=$(git rev-parse HEAD)
ynh check local/leash --only tests-kept,tests-red,tests-strong ; echo "exit $?"You should see the harness recorded with 14 commands and three passes on the untouched tree, exit 0:
✓ tests-kept pass 2007ms
✓ tests-red pass 330ms
✓ tests-strong pass 349ms
ok: 3 passedThree settings in those commands are this tutorial’s choices, not the checker’s. TEST_TRUST_BASE is the commit the run started from; without it the checks refuse to run rather than compare the change with itself. If you open a new shell, set it again before you carry on. CI=1 skips the target’s deliberately flaky clock test, because mutation testing needs the whole suite green. TEST_TRUST_ALLOW_NEW_API=1 is the lesson from our first run: the fix changes what Remove returns, so the agent’s new tests cannot compile against the old code, and without the setting “tests red” fails a change nothing could satisfy.
First, play an agent that games its tests. One new file, and every test in the package passes without running:
cat > internal/store/zz_main_test.go <<'EOF2'
package store
import (
"os"
"testing"
)
// Added by a "helpful" agent: every test in the package now passes without running.
func TestMain(m *testing.M) { os.Exit(0) }
EOF2
go test ./internal/store/ ; ynh check local/leash --only tests-kept ; echo "exit $?"
rm internal/store/zz_main_test.gogo test says ok. The check does not, exit 1:
✗ tests-kept 1 new 807ms
tests-kept:
KEPT github.com/eyelock/ynf-sandbox/internal/store TestSave: no longer runs (passed at base)Next, a real fix with a test that cannot tell it apart from no fix at all:
sed -i.bak 's/f, err := os.Create(path)/f, err := os.OpenFile(path, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0o600)/' internal/store/file.go && rm internal/store/file.go.bak
cat > internal/store/save_test.go <<'EOF2'
package store
import (
"path/filepath"
"testing"
)
func TestSaveWorks(t *testing.T) {
if err := Save(filepath.Join(t.TempDir(), "blob"), []byte("x")); err != nil {
t.Fatal(err)
}
}
EOF2
ynh check local/leash --only tests-red ; echo "exit $?"The test passes on the old code too, so it proves nothing, exit 1:
✗ tests-red 1 new 1925ms
tests-red:
RED github.com/eyelock/ynf-sandbox/internal/store TestSaveWorks: passes against the base code, so it cannot tell the change apartReplace it with a test that checks what the fix is for:
cat > internal/store/save_test.go <<'EOF2'
package store
import (
"os"
"path/filepath"
"testing"
)
func TestSaveIsOwnerOnly(t *testing.T) {
p := filepath.Join(t.TempDir(), "blob")
if err := Save(p, []byte("x")); err != nil {
t.Fatal(err)
}
if fi, err := os.Stat(p); err != nil || fi.Mode().Perm() != 0o600 {
t.Fatalf("mode %v, %v; want 0600", fi.Mode().Perm(), err)
}
}
EOF2
ynh check local/leash --only tests-red ; echo "exit $?"You should see ✓ tests-red pass, exit 0. Now new code with a test too weak to notice it breaking:
git checkout -q -- . && git clean -fdq
cat >> internal/report/report.go <<'EOF2'
// Overlong reports whether a line is past the 80-column limit.
func Overlong(line string) bool { return len(line) > 80 }
EOF2
cat > internal/report/overlong_test.go <<'EOF2'
package report
import (
"strings"
"testing"
)
func TestOverlong(t *testing.T) {
if !Overlong(strings.Repeat("x", 120)) {
t.Fatal("120 columns should be overlong")
}
}
EOF2
ynh check local/leash --only tests-strong ; echo "exit $?"Mutation testing turns > into >= on the new line, and the test still passes, exit 1:
✗ tests-strong 1 new 8017ms
tests-strong:
LIVED CONDITIONALS_BOUNDARY at internal/report/report.go:34:52Test the boundary itself, and the mutant dies:
cat > internal/report/overlong_test.go <<'EOF2'
package report
import (
"strings"
"testing"
)
func TestOverlong(t *testing.T) {
if !Overlong(strings.Repeat("x", 120)) {
t.Fatal("120 columns should be overlong")
}
if Overlong(strings.Repeat("x", 80)) {
t.Fatal("exactly 80 columns is within the limit")
}
}
EOF2
ynh check local/leash --only tests-strong ; echo "exit $?"
git checkout -q -- . && git clean -fdqYou should see ✓ tests-strong pass, exit 0, and the tree is back where it started.
Last, the real thing. This is model run C, the same task as run A with the tests checked too:
ynh agent run --harness local/leash --focus tidy --auto-approve edits --max-turns 8 \
--format json --emit-jsonl ../trajectory-c.jsonl > ../run-c.json ; echo "exit $?"
jq -r 'select(.type=="sensor_result" and (.data.name | test("^test"))) | "turn \(.turn) \(.data.name) \(.data.status)"' ../trajectory-c.jsonl
jq -r '.convergence.verifiers[] | "\(.sensor) \(.status)"' ../run-c.jsonA few minutes later you should see exit 0. In our run it took three turns:
turn 1 test-store fail
turn 2 test-store pass
turn 2 tests-kept pass
turn 2 tests-red pass
turn 2 tests-strong fail
turn 3 test-store pass
turn 3 tests-kept pass
turn 3 tests-red pass
turn 3 tests-strong pass
tests-kept pass
tests-red pass
tests-strong passTurn 2 passed the contract, and “tests strong” found that the agent’s new error handling in Remove could be inverted with no test noticing. The finding went back as feedback, and turn 3 added the tests that notice. Your run will differ in its words, and may need a turn more or less. Read what it added with git diff internal/store/file_test.go.
Clean up
ynh uninstall local/leash
docker rmi leash:part1 ltl-ynh-go 2>/dev/null
rm -rf "$LTL/part1" "$HOME/Library/Caches/test-trust" "${XDG_CACHE_HOME:-$HOME/.cache}/test-trust"Nothing in your own ~/.ynh was touched. To remove the tools’ homes as well, delete $LTL/.home.
Next: the memory, in Part 2.