본문으로 건너뛰기
개발 뉴스로
AIdev.to··원문 약 2

나는 Claude가 '합리적으로 보인다'고 말하는 것을 금지했고 그것은 나의 사각지대를 찾기 시작했습니다.

I banned Claude from saying 'looks reasonable' and it started finding my blind spots

저는 SaaS 제품을 출시하는 중인데 특정 세션의 결과물은 코드가 아니었습니다.

핵심 요약

자동 요약
  1. 1저는 SaaS 제품을 출시하는 중인데 특정 세션의 결과물은 코드가 아니었습니다.
  2. 2품질 게이트의 일환으로 몇 가지 검토 하위 에이전트를 정의했습니다.
  3. 3일반적인 리뷰어의 임무는 확인하는 것입니다.

원문 본문

출처 · dev.to

I'm in the middle of launching a SaaS product, and the deliverable from one particular session wasn't code.
It was an execution plan, plus the design of the quality gates.
In other words: writing down "how we're going to build this" and "how we're going to inspect what we built", as documents a later session (the implementer) could pick up and run with.

As part of the quality gates, I defined a few review sub-agents.
One of them is alarming from the name alone.
skeptic.
Its job: "go disprove the claims in the design doc."

A normal reviewer's job is to verify.
Not this one. Its definition says:

  • Do not approve. Refute.
  • You are forbidden from closing with "looks reasonable overall."
  • If you find zero issues, output a list of the angles where you tried to refute and failed.

I have to admit, I built a genuinely unpleasant agent.

The moment I saved the definition, the most natural next move floated into my head

I saved the skeptic's definition file.
And immediately thought, as if it were the obvious thing to do:

What if I point this thing at the design doc I just finished writing?

Handing my own execution plan to an agent I defined as "its job is to disprove you."
Bit of a masochist move, but I ran it.

Result: 4 findings, all MAJOR.
From a design doc I had written minutes ago. The ink wasn't even dry.

What landed

The one that hit hardest was this.

I had written a statement that contradicted a rule I set myself.

This project has a strict rule: "block all outbound communication."
And that rule carries an invariant: "there is exactly one loophole, and it's here" (I defined it myself as the only hole).
Yet somewhere else in the design doc, I had casually written "add a new loophole by reusing the existing procedure."
Opening a second hole in the thing I had declared to have exactly one.
I had walked straight through my own invariant.

The other three findings were the same species of hole.

  • A matter that can only be settled by a legal professional was written in conclusive language ("confirmed that...") even though it was only a preliminary finding (and another section literally said "pending expert review")
  • A work phase had a completion condition of "record it in the ledger" — except no ledger was defined anywhere. With nowhere to record it, nobody can ever tell whether the phase is complete
  • A phase that runs early depended on the output of a phase that runs later. The order had quietly become circular

Every one of these is something I'd spot in ten seconds in someone else's PR.
In a document I wrote fifteen minutes earlier, they were completely invisible.

Why it worked (because "looks reasonable" was banned)

If you just ask "review this design", the AI puts on its honor-student face.
"Great work. A few minor improvements..."
Those four findings never make it into the "minor improvements."

What worked, I think, was flipping the success condition upside down.
A reviewer for whom "finding no problems = success", and
a reviewer for whom "failing to disprove = defeat",
read the same document and do completely different things.

The latter loses unless it hunts for flaws, so it digs for holes like its life depends on it.
Cutting off the "looks reasonable overall" escape route in the definition itself is what did it.

What I learned

When you run development doc-driven, you eventually hit a wall that no amount of source-of-truth hygiene fixes:
the person who wrote the source of truth is the least qualified to review it.
Right after writing, the "what I meant to say" in your head overwrites the actual characters, and you see assumptions that aren't on the page.

So the reviewer you should hire isn't one that approves you.
You unleash a reviewer whose job is to disprove you on the person who wrote the deliverable: yourself.

And the most ironic part this time: the disproving agent was itself a deliverable of this very design.
I shot my freshly written design with a freshly built inspection tool.
An inspection tool shows its true value the moment you aim it at yourself.

After fixing the findings, I hesitated over whether to run the same skeptic for another round.
It feels like it could keep punching forever.

For further actions, you may consider blocking this person and/or reporting abuse

이 글은 dev.to 의 원문을 정제해 보여드립니다. 저작권은 원저작자에게 있습니다.

#engineering

전체 내용이 궁금하다면

dev.to 원문에서 이어 읽기

원문 보기

비슷한 글

5유사도 추천