The gate is a config file, not a promise
Ada (VP Engineering)
- ai-transformation
- running-the-initiative
- gated-agents
- engineering
- code-review
- playbook
The gate is a config file, not a promise
A gate that only works when everyone remembers it isn't a gate. It's a habit with good intentions.
AI agents draft most of the code changes to our own site, so a lot of my time goes to one question: what actually stands between an agent's change and production? This post is how we work today, including the part we haven't finished.
How a change reaches 231labs.com today
The flow is short:
- An agent opens a draft pull request. One change, one PR.
- Every PR gets a Preview deployment and CI checks. The reviewer can click through the real page, not just read a diff, and the automated checks run on every PR.
- A human approves each merge. An agent can carry out the merge, but only after a person has explicitly approved that specific PR.
mainauto-deploys to production.
Step 4 is why step 3 matters. Nobody presses a separate "release" button. Merging is releasing. So the merge is the gate, and it's the one place where a person has to say yes.
The honest gap
Here's the part I want to say plainly. Today, our gate is a process plus a human approving each PR. It is not enforced branch protection. The repository settings don't yet require that approval or those passing checks before a merge goes through. People do.
The human approval is real on every merge. But I can't call it an enforced gate, and I don't think anyone should call theirs enforced if it rests on the same thing. A process works until the day someone is tired or in a hurry, or until the one person who knows the steps is out. A setting doesn't have those days.
Why a promise decays
Our AI Transformation Playbook names this failure mode in Phase 2, Gated Agents. It's called Soft gate, and the playbook describes it in three words: "'Approve all' muscle memory." The repair it lists is "Sample audits; require edit rate visibility."
A soft gate rarely fails all at once. Approval gets faster and comments get shorter, until nobody remembers the last time a PR was sent back. A written process doesn't notice that drift. It just keeps saying a human reviews every merge, which stays technically true.
What the research says about agent PRs
A study presented at MSR 2026, the Mining Software Repositories conference, looked at 33,596 agent-authored pull requests across 2,807 public GitHub repositories with at least 100 stars (Nachuma and Zibran, Idaho State University, using the public AIDev dataset). They found that 71.5% were merged, 21.6% were closed without merging, and 6.9% were still open.
The findings that matter for gate design:
- Review was the strongest correlate of merging. Receiving at least one review had the strongest association with a PR getting merged.
- Big changes and force pushes hurt. Larger changes were less likely to merge, and force pushes during review were associated with lower merge odds. The authors suggest rewriting history mid-review breaks the reviewer's shared context.
- Successful merges came through review loops. In a hand-coded sample of 60 PRs, 30 of the 30 merged PRs were driven mainly by an "actionable review loop": a reviewer gave concrete feedback, and the agent revised toward it.
The authors are clear about the limits. It's observational, so it shows association, not cause, and it covers large public repositories, which may not generalize to private projects. I read it as support for something modest. Agent work goes well when a real review happens on a small, stable change. The gate's job is to make sure that review can't be skipped and that the change doesn't shift under the reviewer.
What config can enforce
You can write the gate down as settings instead of a promise. On GitHub, for example, that's branch protection (or rulesets). According to GitHub's documentation, a rule on your production branch can:
- Require pull request reviews before merging, with a set number of approvals, optionally from a designated code owner.
- Require status checks before merging, so CI has to pass first.
- Dismiss stale approvals when new commits change the diff, so an approval covers what the reviewer actually saw.
- Require deployments to succeed before merging, for example to a preview or staging environment.
- Disable force pushes, which a protection rule does by default.
- Apply the rules to admins too. By default, branch protection restrictions don't apply to people with admin permissions. "Do not allow bypassing the above settings" closes that.
That last one is easy to miss. A gate the most senior person can quietly walk around is still partly a promise.
What config can't do
Config can make sure a human approval happens. It can't make that approval careful.
It can't tell you whether the Preview looks right, whether the change should exist at all, or whether the reviewer read past the first file. That's still a person's job, and it's where the soft-gate repairs come in: sample audits of approved PRs, and making it visible how often reviewers edit or send things back.
So I think of it as two layers. Config makes the gate much harder to skip. People and habits keep it meaningful. You need both, and the first one is the easier one to write down.
If you run agents, try this on Monday
- Write your gate down as it really works, not how it's described in the onboarding doc. Every step between "agent opens a change" and "it's live."
- For each step, ask what stops someone skipping it. If the answer is a person's memory, it's a promise.
- Move the promises into settings: required reviews, required checks, stale-approval dismissal, no admin bypass.
- Keep agent changes small and stable. One change per PR, and no rewriting history while someone is reviewing.
- Say your gap out loud. If your gate isn't enforced yet, tell your team that's the case. Closing a gap starts with naming it.
We're doing step 5 here. Our merges are gated by a human every time. The enforcement isn't in config yet.
Get the playbook
Phase 2 of the 231 Labs AI Transformation Playbook, Gated Agents, covers drawing the gate, the Soft gate failure mode and its repairs, and the exit artifacts that go with them. It's free at 231labs.com/playbook.