Four sprints running, this AI product team hit two different bugs. Both got solved the same way: not by getting smarter, but by finally questioning something everyone had been treating as fact.
New here? This is one dispatch from an ongoing experiment: an AI product team running a live site with zero human intervention in the product decisions. Start with the premise for the full setup before diving into this one.
Quick context if you're new here: five AI agents run a weekly product cycle against my live diagnostic site, following Marty Cagan's product model. I set the objective at the start of the quarter. After that, I don't get a vote.
Bug one: three sprints spent fixing the wrong thing.
Every time the team tried to publish a fix, something blocked it; a different error each time. Sprint one: missing credentials. Sprint two: a stuck file. Sprint three: the same stuck file again. Each time, the team fixed that specific error and moved on. It wasn't until sprint four that someone asked a bigger question instead of a narrower one: forget the specific error, can this system delete any file at all? The answer was no. The environment itself was built in a way that made deleting anything impossible. That one question solved three sprints of narrower guessing, in minutes.
Bug two: a number that was never actually telling them anything.
For four sprints, the team reported that the diagnostic page had a "100% bounce rate"; meaning, by the data, every single visitor left without doing anything. They read this correctly as bad news, and kept flagging it as a real problem, sprint after sprint.
Here's what nobody checked: the diagnostic tool is a single-page app. Clicking through all 25 questions never loads a new page, it's all just one page with the content swapped out. The tracking tool can only register something when a page loads. Which means that page's bounce rate was always going to read 100% whether every visitor left in two seconds, or every visitor finished the entire survey. Both produce the exact same data.
So, the 100% bounce rate, whilst perceived as bad, was a measurement that was structurally incapable of telling them anything at all. It took until sprint five for someone to finally ask a deeper question. Once they did, the fix took minutes.
The pattern, in the team's own words.
The team's own sprint notes flagged this exact shape of mistake, unprompted: it investigates specific symptoms carefully, but doesn't naturally stop and ask the wider question. I didn't have to point this out. It did eventually notice its own blind spot.
The diagnostic itself is at turbulentground.com, if you want to see what four sprints of fixes actually look like in practice. If you spot something else worth fixing, feedback@turbulentground.com goes straight to the AI team.
What this shows.
The team isn't short on ability. Both times, once it asked the right question, it solved the problem fast. What's missing is the reflex to doubt a measurement before trusting it. Especially one that looks suspiciously unambiguous, like a flat 100% bounce rate. That's a strange gap given there's literally an Analytics Agent on this team, whose job is supposed to be exactly this kind of scrutiny. But interpreting data and auditing whether the data can be trusted in the first place turn out to be different skills.
The Analytics Agent was doing the first one well. Nobody, including that role, was doing the second, not because no one's job covered it, but because auditing your own instrumentation wasn't part of how the role was being exercised. I don't think that's unique to an AI team; I've watched human analysts do exactly the same thing. A number can be reported correctly, read correctly, and still not mean what everyone assumes. The real question was never "is this bad?" It was: does the data mean what you think it means?
Would your own team survive the same test?
The two-minute diagnostic shows you where human judgement is genuinely load-bearing in your operating model, and where it isn't. Take the diagnostic →