Does "loop engineering" translate to Product Management work?

Building a self-drive agent that distills a #1 customer opportunity from transcripts. On definitions of done; verification vs. validation; good data collection

The creme de la creme of my engineering buddies is going all in on “loop engineering” and “autonomy sliders”. I want to stay out of the mud slinging fight about which words to use, my interpretation is: “What do I put in place so I can take myself out of the loop, and let my agents keep going until the output meets a definition of done I set upfront?”

As a PM, I feel like I’m missing out on all that fun. Here I am, manually invoking product discovery skills, and checking every line of output. Like a total noob.

I tell myself: “Product Management work takes human empathy, taste, judgement, stuff AI can’t learn.” But is that a cop out, because I’m simply unable to spell out my definition of done? (and maybe a little bit because I want to keep my job)

Can I imagine an AI that autonomously goes through a bunch of interview transcripts, collects customer opportunities, clusters those, and picks “the right” #1 opportunity to focus on - with the same level of certainty of an autonomous coding loop saying “this feature is now done”?

I’ve spent years manually doing this work and training others. I’ve spent some months building my product discovery skills to be a synthetic second set of eyes (Github URL: fork them, use them!), so I’ve got some of my rules already spelled out. I dove into the ICP prescreen, the opportunity analyst, and the opportunity clusterer and sizer in previous articles.

To me, distilling the #1 opportunity (pain point, wish, desire - a la Teresa Torres) from a wealth of qualitative data sits at the heart of Product Management work. A perfect use case to play around with.

To check its own work, an agent (or loop of agents) needs two ingredients: a definition of done, and verifiers (the checks it runs along the way to make sure it’s progressing towards Done, plus the tools to run those checks). They’re a cascade: the definition of done sets the scope for the verifiers.

Definition of Done

This is the core, it’s what the agent checks itself against to decide it’s finished.

So for an agent whose job is to pick “the right” #1 customer opportunity, what does “right” mean? Two readings:

1. Did we pick the right thing (validation). If we build a solution around this opportunity, do people use and love it.

2. Did we do the work right (verification). Did we follow our rules correctly to reach a defensible # 1 opportunity.

The first reading is the one I actually care about, and my instinct was to throw it out immediately: the answer only shows up months after we ship, and a loop can’t sit around waiting months. But that instinct is wrong. Product teams already run validation inside a loop, they just don’t call it that: ship the change to 10% of users behind a feature flag, watch the conversion metric, roll back automatically if it drops. That signal takes days or weeks to arrive, and that’s absolutely no blocker for an agentic loop. It just wakes up, checks, and acts.

So time was never the barrier. Attribution is: can you trace the result back to the decision? The A/B test buys attribution for software, randomization guarantees the metric moved because of this change and nothing else. No pesky external factors.

My “#1 customer opportunity” has no equivalent. Between “this was the right opportunity” and “a product with traction” sit the solution choice, execution, go-to-market, timing, and many other unknowns. Even when the traction answer finally arrives, it can’t tell me whether the #1 opportunity I selected was the problem.

So my definition of done goes back to the second reading: a defensible #1 opportunity. Software makes exactly this fallback everywhere it can’t afford attribution, which is almost everywhere: “it works, it passes the tests” says nothing about whether the feature was worth building. I’m making the same forced move.

Here are my rules for choosing the “defensible #1 opportunity”:

• every opportunity it weighed was pulled from a real customer quote, nothing invented,

• the importance and prevalence behind each opportunity are backed by the evidence, not guessed,

• the pick wins under my codified selection rule: keep only the pains that average at least four out of five on importance, then pick the most prevalent one. The importance floor comes first, so a shallow pain lots of people mention can’t beat a deep pain fewer people mention (I explained my reasoning in depth in this article).

A definition of done like this is really easy to game. If the rules behind “defensible” are weak, the agent can pass every one of them and still hand me a bad opportunity. A check is never stronger than the rule it rests on. I’ll come back to that later (hello Goodhart’s law!).

Once the finish line is “did we do the work right”, every check underneath it becomes a “did we do this step right” check. Which brings us to the checks themselves.

Verifiers / checks

These are the checks the agent runs between steps, during a step, and at the very end, to tell whether it’s still moving toward the definition of done.

Each check is one of two kinds, and which kind it is decides how much I can walk away.

A deterministic check compares the work against something concrete and gives the same answer every time. A script runs it, the answer is a clean yes or no, and no opinion is involved. Does this quote appear word for word in the transcript, does the score sit above the threshold.

A semantic check needs an AI to read meaning or judge quality. There is no concrete thing to compare against, so its answer can shift between runs and it can be wrong. Is this actually a customer pain or just an offhand comment, do these two opportunities describe the same underlying need. This is where humans fall into endless debates, and AI follows suit.

The difference decides my day. A loop can run itself quite neatly on deterministic checks, because it can trust them and move on. Every semantic check is a spot where the loop is interpreting, and where I still have to look before I believe the result.

The checks in an agentic engineering loop

In an agentic coding loop, almost every check is deterministic.

• Static checks (the type checker, the linter) flag broken code after every edit.

• Unit tests confirm each small piece does what its spec says.

• Integration and end-to-end tests confirm the pieces work together, and a tool like Playwright drives the real running app the way a user would.

Only at the very end do the semantic checks enter, and only for the soft stuff:

• Does the screen look right, is the code well structured, is this a sensible approach.

So the whole stack is deterministic, with a thin layer of semantic judgment at the top. That is what lets a coding loop run on its own for so long, it can trust its deterministic checks and keep going, and the semantic layer is small enough to leave for a human glance at the end.

You might notice the A/B loop from earlier is missing from this list. That’s because agentic engineering loops today stop at the tests: nobody I know feeds the experiment results back in as a check the loop waits on. I personally think that’s the obvious next step, “does the feature move the user behavior metric” as the loop’s final check, validation instead of just verification. But hey, maybe that’s just me.

The checks in my discovery pipeline

My discovery pipeline is the polar opposite. There are some deterministic steps, but most of the work is semantic.

Here’s the pipeline of my Opportunity Selection agent (rules borrowed from my Product Discovery skills), with one example of each kind of check per step (there are more, in the skills themselves, grab them from Github).

Step 1, filter down to the interviews from people who match our ICP. The check compares what we recorded about each person against our ICP definition. If the ICP is loose, deciding who fits is interpretation, a semantic check. If the ICP is explicit, say “developers who live in the terminal instead of the IDE”, and we asked exactly that in the interview, matching becomes a lookup, a deterministic check. One upstream choice decides which camp this check lands in.

Step 2, map the customer journey into phases, the moments in time that opportunities attach to. This one stays largely semantic and human driven, and I’m OK with that. It is the architecture of the work, the equivalent of the brainstorming and planning an engineer does with the AI before the autonomous loop kicks off. The check here, do these phases match how customers actually experience the job, is a judgment call, and I do not expect to hand it off.

Step 3, pull the customer opportunities out of each transcript. What even counts as a customer opportunity is itself a judgment, so I spelled it out as a rule in the skill [screenshot of the rule]. Humans find this really hard, so does AI.

From there, a deterministic check: the supporting quote actually appears in the transcript, word for word, which catches the AI inventing evidence. And a semantic check: is this a real opportunity, or just a feature request, or an offhand remark. I know that humans are quite bad at this, so I don’t trust my AI to do a great job at it either (but hey - building for the models of the future!).

Step 4, cluster the opportunities. Customer A said one thing, customer B said another, are those the same opportunity or two different ones? A deterministic check: every opportunity in a cluster traces back to a real quote someone actually said. A semantic check: the same or different opportunity? This one’s also really tough.

Step 5, score, rank, and pick the number one. This step is deterministic, with a big asterisk.

I have this strong belief that I’d rather solve a hair on fire problem shared by few, than a nice-to-have shared by many. This translates into my rule that first I filter for opportunities with a median score of at least 4 out of 5 (importance as a floor), and out of those pick the one shared by the most interviewees (prevalence as a decider.,

The math either puts something on top or it does not. The asterisk: this check only confirms that the calculation was done correctly. The importance and prevalence numbers it calculates with came out of the semantic steps before it.

Hold the two lists side by side. The agentic engineering loop runs on a deterministic stack and saves its thin semantic layer for the very end. My discovery pipeline is the inverse: a few deterministic checks inside work that is mostly semantic, and semantic checks are error-prone and unreliable. That’s why taking myself out of the loop feels irresponsible. Every semantic check is a place where the loop is guessing, and mine don’t sit at the end as a final glance, they sit at the start and in the middle, where every later step builds on their output.

Measuring how much of the pipeline is guesswork

But I don’t have to guess how much of my pipeline is guesswork, I can measure it. Run the whole pipeline on the same transcripts five times. Deterministic checks return the same answer every run, semantic checks don’t. So if the five runs produce different #1 opportunities, the difference was caused by the semantic steps. Count how many runs agree on the same #1, and you have a pretty good measure of how reliable the pipeline is. This is also why the measurement has to cover the whole pipeline and not just one step: step 5 returns the same ranking every time you feed it the same numbers, but the semantic steps before it feed it different numbers on every run.

In my experiment, I kept the same phase map for every run - this is the output of my planning phase, but re-ran opportunity-analyst, opportunity-clusterer, and opportunity-sizer 5 times. Each run came up with slightly different variations on the same #1 opportunity (“I can’t tell which companies are genuinely AI-native from the outside”), belonging to the same phase. Pretty good!

Moving checks to the deterministic side

So the real question is: how do I move checks from the semantic side to the deterministic side? There are two levers.

The first lever, codify the rule into math. Step 5 already does this. My selection rule says: keep only the pains that average at least four out of five on importance, then pick the most prevalent one out of those. Notice that this rule has exactly the same shape as the rollback rule from the A/B loop: roll back if the uplift sits below n percent. In both cases, someone picked a number they bet correlates with “good”, and from then on the loop just runs the comparison.

But the two rules hide very different amounts of judgment. Every codified rule contains two choices: which quantity to compute, and where to set the bar. For the rollback rule, the quantity is conversion, which is almost the outcome itself, and the experiment proves the connection, so the only guess left is the threshold. For my rule, the quantity is importance and prevalence, and the connection to the outcome, people use and love what we build, is itself a guess that nothing tests.

This is Goodhart’s law territory: the weaker that connection, the more room the agent has to pass every check and still hand me a bad opportunity. The validation question I threw out of the definition of done climbs right back in through the rule I chose. Codifying a rule doesn’t remove the judgment, it concentrates all of it into one question: is this the right rule?

The second lever, capture the data the math needs, so the AI doesn’t have to infer it. That’s step 3: instead of reading importance out of someone using words like “hate” or “despise”, ask them to rate importance from one to five, and ask when it last happened/how often it happens. A shaky inference becomes a recorded number.

Some checks won’t move by either lever, and I’m honestly not sure yet how many. Mapping the journey into phases (step 2) is planning work. I can’t imagine how I can move “Are these two opportunities the same or distinct” to the deterministic side, but maybe I lack imagination. For now that’s still the part of the job I’m not walking away from.

Where this leaves me

Professional - I honestly don’t know man, But let’s merrily jump over that point 🤷‍♀️

Now, for some useful take aways: I can’t attribute traction back to my pick, so Done means a defensible #1 customer opportunity. A defensible opportunity only holds if the rules underneath it are good. And the rules only work if the data feeding them was captured, on purpose, while we were still talking to the customer.

That makes the interview guide the spec for discovery. Ask for the importance rating and the last occurrence during the interview, and scoring becomes math instead of tone-reading. Ask the explicit ICP questions upfront, and the ICP filter becomes a lookup. Every piece of “what good looks like” I pin down while I’m still talking to the customer is one more check that can run without me. Collecting data explicitly moves checks from semantic to deterministic.

Engineers got their autonomous loops off the back of decades of compilers, test frameworks, and specs. I think ours starts with a better interview guide.