How Building an AI Harness Created Organizational Loops

Building reliable AI systems naturally creates organizational loops

Hello, I’m yeTi.

Recently, I have been seeing the word “loop” more often in discussions around AI agents.

Reflection.
Self-Refine.
Self-improving agents.
Loop Engineering.

The underlying idea is usually similar.

Instead of expecting an AI agent to produce the right answer in a single attempt, we let it inspect its own output, identify what is missing, fix the result, and try again.

At first, this felt natural to me.

If AI agents are going to be useful in real software engineering, they need some kind of iterative structure.

But while building and operating an AI engineering workflow for SQLGen, I started to see loops differently.

I did not start by trying to design loops.

The goal was not “Loop Engineering.”

The goal was reliability.

I could not fully trust the output produced by AI, so I added validation.

Validation failed, so I added recovery.

Recovery happened repeatedly, so I added state management.

As the number of states grew, I needed review and approval flows.

Local validation passed, but the system still failed in the development environment, so I added E2E validation.

And as failures accumulated, I eventually needed a mechanism to analyze failures themselves.

Only after all of this did I look back and realize something.

Every reliability mechanism I had added had created a loop.

That is why I now understand loops a little differently.

A loop is not always something we design first.

Sometimes, it is the shape a system naturally takes when it keeps trying to become reliable.

What I wanted to build was an AI engineering organization

The project I am building is called sqlgen-ai.

It is an AI-driven engineering workflow for the SQLGen platform.

From the beginning, the goal was not to build a simple AI coding assistant.

If the goal had only been to generate code faster, using ChatGPT or Codex more effectively might have been enough.

But what I wanted was a larger system.

A system that could interpret requirements, write an implementation plan, modify code, run tests, create a Merge Request, receive review feedback, validate the result in a development environment, analyze failures, and create follow-up fixes when necessary.

In other words, I was trying to model a software development process where AI agents take on specialized organizational roles.

So I started treating AI less like a tool and more like a set of organizational members.

The PM Agent writes the plan.

The Developer Agent implements the change.

The Validator Agent verifies the result.

The Review Triage Agent responds to review feedback.

The E2E Validator Agent checks the behavior in a real development environment.

GitLab became the coordination layer for these agents.

Issue and Merge Request labels were no longer just tags.

They became state transitions for the organization.

issue:ready-for-review
issue:approved
dev:ready
dev:running
dev:done

mr:ready-for-review
mr:approved
mr:review-rejected

e2e:ready
e2e:running
e2e:done
e2e:failed

Even at this point, I was not thinking about loops.

What I needed first was a controllable execution flow.

I wanted AI agents to act only when the system was in the right state.

I wanted every action to leave evidence behind.

I wanted the workflow to move forward only when the previous step had been verified.

Looking back, this was the beginning of the harness.

The harness was not a way to trust AI

One of the most dangerous moments when using AI coding tools is when the output looks convincing.

The code is generated.

The explanation sounds reasonable.

The agent may even say that it ran the tests.

But in practice, tests may still be failing.

Types may be wrong.

Part of the requirement may have been missed.

Or the implementation may quietly damage the existing design.

So I decided not to trust the AI’s claims.

Instead, I required evidence.

“Implementation is done” is not evidence.

There must be test results.

“Review is complete” is not evidence.

There must be an approval label.

“E2E looks fine” is not evidence.

The scenario must pass in the development environment.

From this perspective, the harness was not a mechanism for trusting AI more.

It was a mechanism that made it possible to operate the system without trusting AI too much.

No matter what the AI says, the workflow can only move forward when the required state and evidence exist.

At that moment, the workflow stopped being simple automation.

It started to become an operating system for AI engineering.

The first loop emerged from plan review

The first workflow was simple.

The PM Agent created a GitLab Issue and a plan.md.

It interpreted the requirement, wrote an implementation plan, and included E2E scenarios when needed.

A human reviewed the plan and added the issue:ready-for-review label.

Then the Plan Review Bot reviewed the plan.

If the plan was approved, the issue received issue:approved.

If the plan was rejected, it received issue:review-rejected.

At first, this rejection required a human to intervene again.

Someone had to read the review comments, revise the plan, and request another review.

But the same kinds of feedback started to repeat.

The plan was incomplete.

The Acceptance Criteria were unclear.

The E2E scenario was missing.

The implementation scope was too large.

These were real issues, but they were not issues that required a human every time.

So I built the PM Review Triage Agent.

This agent detects issue:review-rejected, collects the rejection feedback from the issue comments, resumes the previous PM session, revises the plan, updates the issue description, and attaches issue:ready-for-review again.

Structurally, it looks like this.

Plan
↓
Review
↓
Rejected
↓
Revise Plan
↓
Review

This is clearly a loop.

But it was not designed as a loop from the beginning.

It emerged because plan review kept failing in predictable ways, and I needed a recovery path.

The second loop emerged from implementation validation

The next problem appeared during implementation.

The Developer Agent used Codex to implement the change.

It analyzed the codebase using the PM Agent’s plan and applied the required modifications.

At first, it looked like implementation could end there.

But in real operation, it did not.

The generated code failed ruff.

It failed mypy.

Delta-based pytest broke.

Sometimes the implementation looked correct, but it did not fully satisfy the Acceptance Criteria.

So I added a local validator.

After the Developer Agent completed the implementation, the validator ran checks against the changed files.

Implement
↓
Local Validate

But once the validator existed, another problem appeared.

When the validator returned NO-GO, a human still had to fix the code.

That created too much operational overhead.

Especially for failures with clear feedback, such as lint errors, type errors, or simple test failures, the agent could often fix the problem by itself.

So I added automatic fix retries.

But I did not allow infinite retries.

The agent could attempt automatic fixes up to two times.

If it still failed after that, the workflow was marked as failed.

Implement
↓
Validate
↓
NO-GO
↓
Fix
↓
Validate
↓
NO-GO
↓
Fix
↓
Validate
↓
Fail

The important part here is not the retry itself.

The important part is the boundary.

An infinite loop is not reliability.

It can become a way to hide failure.

So every loop needs an exit condition.

The system has to distinguish between what the AI can reasonably fix by itself and what requires human intervention.

The third loop emerged from code review

Even after local validation passed, the Merge Request could still be rejected.

This showed me an important distinction.

A validator catches mechanically verifiable problems.

Lint.

Types.

Tests.

Local behavior.

But review is different.

Review is closer to an organizational judgment.

Does this code fit the existing architecture?

Is the scope appropriate?

Do the names communicate intent?

Are the tests meaningful?

Will this be maintainable later?

These questions cannot be reduced to a passing test suite.

So the Review Bot reviews the Merge Request.

If the review is rejected, the MR receives the mr:review-rejected label.

Then the MR Review Triage Agent detects that state, collects the review comments, resumes the previous implementation session when a code change is needed, or posts a response when clarification is enough.

Then the MR goes back into the review flow.

MR
↓
Review
↓
Rejected
↓
Triage
↓
Fix or Respond
↓
Review

This loop has a different nature from the validation loop.

The validation loop checks whether the code works.

The review loop checks whether the code fits the organization’s standards.

That difference made the idea of an AI engineering organization feel more concrete to me.

An agent that generates code is not an organization.

An organization needs standards.

And when work does not meet those standards, there must be a path that brings it back.

The fourth loop emerged from E2E validation

Local validation and review were still not enough.

Some changes passed locally but failed in the development environment.

That is where another class of problems appeared.

External API connections.

Environment variables.

Real data conditions.

Browser or endpoint behavior.

Interactions across multiple components.

These were difficult to verify with local tests alone.

So I added an E2E Validator.

When the GitLab CI pipeline succeeds on the develop branch, the MR becomes e2e:ready.

The E2E Picker detects that state.

The E2E Validator creates a worktree for the MR branch, routes E2E scenarios based on the changed files, and runs smoke tests against the development environment.

If the tests pass, the MR becomes e2e:done.

If they fail, Codex analyzes the failure logs and posts an analysis comment on the MR.

When necessary, the system creates a fix MR and sends it back through validation.

Deploy
↓
E2E
↓
Failed
↓
Analyze
↓
Fix MR
↓
Validate

At this stage, the loop was no longer a single agent correcting itself.

The loop involved the deployment environment, CI, GitLab labels, Merge Requests, E2E scenarios, and failure analysis agents.

It became difficult to call this a simple agent loop.

It was closer to an organizational loop.

The fifth loop emerged from learning failures themselves

More recently, I added a Hard Fail Analyzer.

This mechanism runs when the Developer Agent workflow terminates abnormally.

For example, the system may produce states such as:

ABORT_DIRTY_WORKSPACE
ABORT_SSH_FAILED
PRE_IMPL_TIMEOUT
PRE_IMPL_FAILED
MR_FAILED
INLINE_VALIDATION_NO_GO

Previously, when these failures occurred, a human had to inspect the logs and identify the cause.

But repeated failures eventually become patterns.

So I started leaving failure state files, tracking the related logs, and asking Claude CLI to analyze the final portion of the log.

The Hard Fail Analyzer summarizes the root cause.

It decides whether retry is feasible.

It suggests possible workflow improvements.

Hard Fail
↓
Log Analysis
↓
Root Cause
↓
Retry Feasibility
↓
Workflow Improvement Suggestion

This step is interesting because the loop is no longer only about fixing code.

Earlier loops improved artifacts.

They revised plans.

They fixed code.

They responded to review comments.

They repaired E2E failures.

But the Hard Fail Analyzer treats the workflow itself as the object of improvement.

It does not simply ask, “How do we fix this code?”

It asks, “Why does this kind of failure keep happening, and how should the workflow change?”

This changed the question I was asking.

At first, I was asking:

“How can AI generate better code?”

Now I am asking:

“How can an AI engineering organization learn faster from failure?”

That difference feels important.

Patterns are discovered before they are applied

At this point, I was reminded of an older idea in software engineering.

The GoF design patterns.

Many developers first learn design patterns as something to apply.

If you use Factory, Strategy, Observer, or Adapter, your design may feel more sophisticated.

I also used to think that way.

But after spending enough time in real projects, my view changed.

A pattern is not something that should be applied before the problem is understood.

A pattern is something discovered from repeated problem-solving experience.

Object creation becomes increasingly complex.

So we separate the responsibility for creation.

Later, we realize that the structure resembles Factory.

Conditional logic keeps growing.

So we separate behavior into interchangeable objects.

Later, we realize that the structure resembles Strategy.

State changes need to be propagated to multiple objects.

So we introduce observers.

Later, we realize that the structure resembles Observer.

The order matters.

First, there is a recurring problem.

Then there is a repeated solution.

Then we give that solution a name.

But in practice, this order is often reversed.

Developers sometimes try to apply patterns before they have fully understood the problem.

When that happens, a pattern no longer simplifies design.

It becomes decoration.

I think Loop Engineering has a similar risk.

If we try to apply a loop first, we may only create a repetitive process.

But when we detect failure, define validation criteria, create recovery paths, and verify the result again, we may eventually discover that the structure we built was a loop.

In that sense, the loops I found in sqlgen-ai are similar to design patterns.

They are not techniques I applied from the outside.

They are names for operational structures that emerged from repeated reliability problems.

That is why I now understand loops less as a trend and more as an operational pattern for reliable AI systems.

There was not one loop

It is easy to imagine a single loop.

Generate
↓
Review
↓
Fix
↓
Retry

But the structure that emerged in actual operation was not that simple.

Inside sqlgen-ai, there are multiple kinds of loops.

There is a Plan Loop that improves implementation plans.

There is a Validation Loop that makes local checks pass.

There is a Review Loop that incorporates reviewer feedback.

There is an E2E Loop that verifies real behavior in the development environment.

There is a Failure Learning Loop that analyzes abnormal workflow failures and suggests improvements.

These loops handle different kinds of failure.

Planning failure.

Implementation failure.

Quality failure.

Environment failure.

Operational failure.

They are not the same problem.

So they cannot be solved by one universal loop.

Each failure point needs its own validation method and recovery path.

From this perspective, the core of Loop Engineering is not writing a loop.

It is identifying where failure happens,

defining how that failure is detected,

deciding what kind of recovery is allowed,

and sending the system back through validation.

In other words, before there is a loop, there has to be a harness.

The harness comes first, and the loop follows

The way I understand it now, a harness is an execution structure that allows AI agents to work safely.

More concretely, it needs four things.

First, it needs state.

The agent must know what it is allowed to do now.

In sqlgen-ai, GitLab labels play this role.

Second, it needs boundaries.

The agent must know how far it can go and when it must stop.

For example, automatic fixes are limited to two attempts.

Final merge remains a human decision.

Third, it needs evidence.

The next step should be based on validation results, not on the AI’s explanation.

Test logs, review approval, E2E results, and failure analysis comments are all forms of evidence.

Fourth, it needs recovery paths.

When something fails, the system should not always require a human to restart the process from the beginning.

Within a safe boundary, the system should be able to recover and try again.

Once these four elements exist, a loop naturally appears.

State
↓
Action
↓
Evidence
↓
Recovery
↓
State

That is why I think starting with the loop can be misleading.

A loop without clear validation is just repetition.

A loop without an exit condition can become a failure amplifier.

A loop without ownership can blur responsibility.

A loop without evidence can strengthen the AI’s own confidence without improving the system.

Reliable loops require a harness first.

It was closer to Reliability Engineering than Loop Engineering

New terms appear quickly in the AI field.

Prompt Engineering.

Context Engineering.

Agent Engineering.

Harness Engineering.

Loop Engineering.

Each term points to something useful.

But if I describe my own experience, the sequence was different.

I did not start from prompt optimization.

I did not stop at adding more context.

I did not begin by designing a beautiful loop.

The actual sequence was closer to this:

AI fails
↓
Failure must be detected
↓
Validation criteria are added
↓
Recovery paths are added
↓
State transitions are added
↓
The system runs again
↓
Repeated failures become learning signals

If I had to name this, I would call it Reliability Engineering.

It is not about making AI smarter.

It is about making sure the system does not collapse when AI is wrong.

It is about building a workflow that can decide what to do next even when the AI makes mistakes.

And in that process, loops emerged.

Closing thoughts

If I had started by trying to design loops, I do not think I would have built the structure I have now.

I probably would have created a simple iterative process.

Generate.

Review.

Fix.

Generate again.

But real operation required something more specific.

I needed a loop for failed plans.

I needed a loop for broken implementations.

I needed a loop for rejected reviews.

I needed a loop for development environment failures.

I needed a learning loop for workflow failures.

In the end, loops were not a single technique.

They were operational patterns discovered through the repeated pursuit of reliability.

I now summarize the process like this.

Repeated Failure
↓
Operational Response
↓
Reusable Structure
↓
Named Pattern

And from the perspective of AI systems, I would describe it like this.

Reliability
↓
Harness
↓
Validation
↓
Recovery
↓
Organizational Loop

Loop was not the goal.

Loop was the structure that emerged while building a harness.

Just as GoF design patterns gave names to recurring design experience, I think loops in AI engineering are names for recurring operational structures found while building reliable systems.

And if AI coding agents evolve into AI engineering organizations, those organizations will eventually need multiple loops.

Not because loops are fashionable.

But because a reliable organization must be able to detect failure, leave evidence, recover safely, validate again, and learn from what failed.

That, to me, is the most practical meaning of Loop Engineering in reliable AI systems.

But there is one more question this raises.

If loops emerge naturally from reliability work, should every failure become a loop?

I do not think so.

Some failures should be retried.

Some failures should be recovered automatically.

But some failures should stop the system immediately.

So the next question is not how to create more loops.

The next question is how to design stop conditions for AI workflows.

A reliable AI system does not only know how to retry.

It also knows when to stop.