"Unable to use the delete button to remove an item from the shopping cart."
That line was in the brief for problem #4 at the buildX hackathon: build an agent that tests web workflows. By the end of the first build day it was bug B5 in a small fake shop I'd made, and my agent found it on its own.
I built test-badger in five days, with Claude Code as a pair programmer. The case study is on my works page. This is just a few notes on the week, mostly about the things I got wrong.
I started by writing things down
Before any code, I spent a day writing nine short documents: what the product is, every screen, how the agent works, the demo shop and its bugs, and a plan with the things I was willing to cut. It felt slow. It turned out to be the most valuable day of the week.
Every decision went into a log the moment it was made, with a reason. When I asked myself "why did we do it this way?" on day five, the answer was one search away.
The decisions came before the build days
Decisions written to the log, and commits pushed, per day.
| Day | Decisions logged | Commits | What happened |
|---|---|---|---|
| Plan · Oct 3–4 | 16 | 2 | 9 spec documents, decision log, working rules |
| Day 1 · Oct 5 | 0 | 8 | skeleton, auth, demo shop, deployed |
| Day 2 · Oct 6 | 0 | 5 | agent end to end, evaluation 9/9 |
| Day 3 · Oct 7 | 6 | 3 | redesign, rename, re-run + compare, scenario video |
| Day 4 · Oct 8 | 3 | 2 | Hard/Easy modes, judges' account |
| Day 5 · Oct 9 | 13 | 19 | budgets, external benchmark, feature freeze, docs, go-public audit |
| Day 6 · Oct 10 | 1 | 2 | redesigned frontend beside the old one |
When it missed
The first proper evaluation found 7 of the 9 bugs I'd planted, and it missed the same two every time. The tempting fix was a longer prompt. Neither real fix touched the prompt: in both cases the agent simply couldn't see something it needed. Once it could, it found both.
Since then, when an agent seems confused, my first question is "what was on its screen?"
A store I didn't build
After 9 of 9 on my own shop, I tried Toolshop, a public practice store whose bug list was written by somebody else. Of the 19 bugs in scope, it found one.
That stung, and it was the most useful hour of the week. The worst finding was my own: with one scenario passed and five inconclusive, my report showed a big green "All passed". The agent had been careful; my summary wasn't. I fixed it the same day.
Your own test site always flatters you. Next time I'd try a stranger's site on day two, not day five.
How it ended
test-badger scored 83 out of 100. The judges ran it themselves and wrote that results from a controlled demo can't necessarily be generalised to all websites. They're right. Toolshop had already shown me the same thing.
What I'd tell myself on day zero
- Write the answer key before the agent. It made every change measurable.
- Try a stranger's site early. Your own always makes you look good.
- Let "I don't know" be an answer. For a testing tool, a confident wrong answer is worse than none.
The badger's slogan is find it first. That week, the agent found most of the bugs in my own process before I did. I'll take that.