Setting up AI code reviews in our operations

As our work involves a substantial amount of software development, we stick to established practices as a proven method. With the rise of AI, we've also looked at embedding it into our workflow. AI is a tool, a very capable one, but still a tool. It can do the work, but the person responsible for that work is still a human. So we use these tools to make our lives easier and let us deliver faster.

One of the bottlenecks in a typical development lifecycle is code review. It's a crucial step that can't be skipped, but optimizing it pays off a lot. One of our internal projects automates code review. People still review everything, the bot doesn't replace that, but adding AI to the loop catches most bugs at an early stage and lets humans spend their attention on the parts that actually need judgment, instead of the mechanical stuff.

We also use AI to write code, and the important part is using a different model for review than for implementation, to cut down on correlated blind spots. Think of it as having a different person review your work, rather than reviewing your own.

Why this matters

• Reviews get slow exactly when the team is busy, which is exactly when bugs are most likely to slip through.
• We aim for small changes, but a large diff is sometimes unavoidable. That's exactly where a bot helps most, catching what a human reviewer's fading attention would otherwise miss.
• It's advisory, not a gate. It never blocks a merge on its own. It exists to shrink the pile a human has to work through, not to replace their sign-off.

Two models, not one

Our reviewer isn't tied to a single vendor. It can run as either the OpenAI Codex CLI or the Claude Code CLI, chosen with one configuration setting: same prompt, same output schema, same posting logic either way, just a different driver script per engine.

In practice we use Claude to write code and Codex to review it. That split is the whole point: a model reviewing its own output tends to agree with itself, rationalizing the same blind spots it had while writing the code. Pointing a different model at the diff is closer to handing your PR to a colleague who wasn't in the room when you wrote it.

Switching engines is a one-line config change on our side, nothing a consuming project has to touch, since the choice is made where the review actually runs, not where the code was written.

What it actually catches

The prompt is deliberately narrow: review only what the diff introduces or makes reachable, not pre-existing issues, not "while we're here" nitpicks, not whether the feature should have been built at all. It's given a full checkout of the branch (not just the raw diff text), so it can open a changed function's callers, its types, its config, and check a claim against the actual code instead of guessing.

Where it helps

• Correctness bugs a careful reviewer would catch on a good day: off-by-one errors, null/undefined dereferences, division by a count that can be zero, inverted comparisons, swallowed exceptions, missing await, leaked handles or listeners.
• Security smells: injection, leaked secrets, missing authorization checks, unsafe input handling.
• Reuse and simplification opportunities, and house-style violations our linter would reject if the diff introduces them.

Where it doesn't

• Whether the feature is the right thing to build, or the right design for it.
• Cross-service architectural trade-offs that live outside the diff entirely.
• Anything that depends on facts the checkout can't verify: whether a pinned dependency version actually exists, whether a third-party library call is well-formed (its own source isn't installed in the review checkout), or what a value looks like at runtime. The prompt explicitly tells the model to stay quiet rather than guess on any of these. A wrong comment costs more trust with the team than a missed one.

Every finding carries a priority (0 for real bugs and security holes, higher numbers for edge cases and maintainability) and a confidence score, so the summary reads as a ranked list rather than a wall of equally-weighted comments.

Isolating it safely

The diff and the branch it came from are untrusted input. A merge request is, by definition, something someone outside the trusted pipeline gets to shape. So the review never runs on the branch under review. It runs as a job on our own protected pipeline, triggered from the project's merge-request pipeline, with only enough information forwarded to fetch the diff over the API: a project ID and an MR number. The trigger token that starts it can do nothing but start that one pipeline. It can't read secrets, and it isn't the token the review job itself uses to post comments.

Once the review job is running with real credentials in hand, the model itself still has to touch the untrusted diff and checkout, so those credentials get stripped from its process environment before it ever runs: the GitLab token used to post comments, the CI job token, the registry password, all of it. Nothing in the diff can read or exfiltrate a secret it was never handed.

The two engines get there differently, and it's worth knowing the difference:

• Codex authenticates from a credential file that's never in its process environment at all, and runs under --sandbox read-only, which blocks filesystem writes and network egress at the OS level.
• Claude Code has no file-based credential form here. Its own token has to live in the process environment for the CLI to authenticate. The mitigation is an explicit tool allowlist: --allowedTools "Read,Glob,Grep" under a non-interactive permission mode, with no Bash, no Edit/Write, no network tools. That's enforcement by the CLI's own permission logic rather than a kernel-level sandbox, a real difference in the trust model, and one we'd rather document than paper over.

Wiring it into the pipeline

A consuming project adds one small job to its merge-request pipeline: it fires in the .post stage, after the project's real tests have already run, so the review can never make an MR look green before its own checks finish, and all it does is start our review pipeline with the MR's project ID and IID. It holds no meaningful secret, just a trigger token scoped to start that one pipeline.

On our side, that triggers the actual review: fetch the MR diff over the API, strip out lockfiles and generated files (they can't produce a legitimate finding, and historically they've eaten over half the token budget on some diffs, so we still list their names so the model knows they changed), fetch a checkout of the branch head for context, run the selected engine, and post results back as inline comments on the exact lines they concern.

One detail we're fairly happy with: the review job also reads back the merge request's own CI results, did the type-check job pass, did the build fail, and feeds that to the model as extra context, but only after verifying that pipeline is genuinely for this MR at this exact head commit. A failed compile job's errors get treated as ground truth; a passing one is only weak corroboration. That stops the model from second-guessing a type error it could just be told about, without letting an untrusted, attacker-shaped CI section override what it reads in the code.

Findings below a confidence threshold get filtered out before posting, and a re-run on the same commit, a retried pipeline, a re-triggered job, is detected and skipped entirely rather than reposting an identical review. If new commits land, the old comments get cleared and the old discussions resolved before the new ones go up, so switching engines or pushing a fix never leaves stale comments sitting alongside fresh ones. Zero findings above the threshold means the bot approves the MR itself; any findings mean it removes its own approval and posts a summary table ranked by priority, so a human reviewer's first look is "what's P0" rather than scrolling a wall of comments.

What it costs, and where it pays off

Both engines bill against a subscription by default, falling back to an API key only if that credential is missing or expires mid-run. So running reviews doesn't add a new per-MR cost, it draws on the same seat we already pay for writing code.

The real cost was in the setup. Getting the isolation model right (sandboxing, credential scrubbing, the auth write-back logic) needed real test coverage before we trusted it with production credentials in a pipeline. Tuning the prompt to stay quiet on unverifiable claims took more iteration than wiring the CI integration did.

We haven't put a number on the payoff. What we can say concretely: every merge request gets read before a human looks at it, the same checklist runs on line 1 and line 400 of the diff, and findings arrive ranked by confidence and priority, so a reviewer's starting point is a short list instead of the raw diff.

It now runs across both our internal projects and customer projects. The value we actually see isn't on a cost line: fewer bugs make it past review into the code we ship, and the time between a feature landing and reaching production shrinks because less of it is spent in a review queue.