PR security review: an AI workflow that reviews risky pull requests

What it costs to run

At 80 pull requests a day, 70% low risk and 30% risky, that is about $7.69 a month. If every one took the risky route it would be $17.27.

Estimated before running, from the prompt each step is sent and an assumed reply length, at list prices for each step's model tier. Want the same numbers for your own agent? Paste its code into the cost calculator.

Sample input

PR #482 in payments-api: "Add refund webhook endpoint". Adds POST /webhooks/refund, touches the auth middleware and a SQL query. 3 files, +142 −18. Security review before merge? --- diff (excerpt) --- +router.post("/webhooks/refund", async (req, res) => { + const sig = req.header("x-signature"); + if (sig != hmac(process.env.WEBHOOK_SECRET, req.rawBody)) return res.sendStatus(401); + const { orderId, amount } = req.body; + const rows = await db.query(`SELECT * FROM orders WHERE id = '${orderId}'`); + logger.info("refund webhook", req.body); + await refunds.create({ orderId, amount }); + res.sendStatus(200); +});

How the workflow runs, step by step

Sample replies show what each step returns for the sample input above. In the editor, Sim shows the same replies for this input without calling a model; Live runs your own models.

1. 1. Start

   Pull request

   The PR title, description and diff start the run. A CI job can send the same text for every pull request.

   Next: Map the change

2. 2. Agent

   Map the change

   A small model lists what changed, what it touches (auth, SQL, secrets, payments) and any new inputs an attacker could reach.

   Sample reply

   FILES: payments-api: a new refund webhook route, an auth middleware change and one SQL query.
   TOUCHES: auth, crypto, sql, payments, input parsing
   NEW INPUTS: POST /webhooks/refund with orderId and amount in the body, and the x-signature header
   ROUTE_HINT: deep review

   Next: Secure coding rules

3. 3. Tool

   Secure coding rules

   Searches your team's secure coding rules for the ones that apply.

   Next: Risk

4. 4. Router

   Risk

   Keyword rules send auth, SQL, token, secret, webhook or payment changes to the deep review, and README, docs or CSS changes to the quick check. Anything unclear gets the deep review.

   Next: Deep review (risky), Quick check (low risk)

5. 5. Agent

   Deep review

   A stronger model reviews the diff against the rules: each finding with its severity, the line, what an attacker could do and the fix, plus tests to add.

   Sample result

   VERDICT: block merge
   FINDINGS:
   • high · SELECT * FROM orders WHERE id = '${orderId}': orderId comes straight from the request body, so a caller can inject SQL. Fix: a parameterised query (WHERE id = $1).
   • high · refunds.create({ orderId, amount }): the refund amount comes from the request, so a replayed or altered call can refund more than was paid. Fix: load the order and refund at most its captured amount.
   • medium · sig != hmac(...): a plain string compare leaks timing. Fix: crypto.timingSafeEqual on equal-length buffers.
   • medium · no timestamp or event id check, so a captured webhook can be replayed. Fix: sign a timestamp, reject anything older than 5 minutes and store processed event ids.
   • low · logger.info("refund webhook", req.body): logs the whole payload on a payment route. Fix: log orderId and the event id only.
   RULES: SQL, Webhooks, Money, Logging
   TESTS TO ADD:
   • An orderId containing a quote returns 400 and runs no query.
   • A refund above the captured amount is rejected.
   • A replayed webhook with the same event id is ignored.
   NOTE: a reviewer approves before this is posted to the PR.

   Next: Reviewer approve

6. 6. Human approval

   Reviewer approve

   The run pauses. A reviewer approves before the review is posted or the merge is blocked.

   Next: Review comment

7. 7. Result

   Review comment

   The approved review, ready to post on the pull request.

8. 8. Agent

   Quick check

   For low-risk changes, a cheap model writes one line on why it is safe and anything to check before merging.

   Next: Low-risk note

9. 9. Result

   Low-risk note

   The low-risk note ends the run. No reviewer time and no strong model.

Why build it as a graph

Running a strong model over every diff is slow and expensive, and most pull requests do not need it. As a graph, a cheap pass maps every change, the strong reviewer only reads the risky ones, and a person stays between the model and the pull request. You can see what each step costs before you wire it into CI.