When we started building PRCheck, we tested everything available. We ran existing AI review tools on real-world pull requests from open-source projects and from teams we had access to. The results were revealing, but not in the way the marketing materials suggested.
The tools were good at a narrow class of problems: stylistic issues, obvious missing null checks, suggestions for shorter code. They were not good at the things that matter most in code review: understanding whether a change is correct given the surrounding codebase, identifying subtle logic bugs introduced by the interaction of modified and unmodified code, and flagging security-sensitive patterns that require understanding data flow rather than surface appearance.
This post is about what we found, why we think it happens, and how we approached building something different.
The diff-in, comment-out problem
Most AI review tools operate by feeding the diff text into a language model and asking it to comment. This works reasonably well for self-contained changes. If a PR adds a new utility function with a bug in it, the model can often identify the bug because all the relevant context is in the diff.
The problem is that self-contained PRs are not the common case. Most PRs modify existing functions, add calls to existing infrastructure, or change behavior in ways that interact with code that did not change. The diff shows the changed lines. It does not show the functions being called by the changed code, the callers of the modified function, or the data structures that the modified code reads and writes.
A diff-based reviewer is working with partial information. It is like reading only the scenes where a character speaks and trying to evaluate whether their behavior is consistent with who they are supposed to be. You might catch obvious mistakes, but you are going to miss anything that requires understanding the larger context.
We had a specific failure case that drove this home early in our development. A PR modified a Python API endpoint handler to add a new query parameter. The diff was clean: a few lines added to extract and validate the parameter, and a few lines to use it in the database query. An AI review of the diff produced no significant findings. The bug was that the new query parameter was being passed to a database utility function that did not sanitize its inputs, because that utility had been written under the assumption that all its inputs came from internal service calls. The vulnerability was in the unmodified code's assumptions about its callers, not in the diff itself.
Why call graph context changes everything
When we analyzed the categories of bugs that escaped AI review tools in our testing, call graph blindness was the dominant pattern. The bugs were not in the changed lines. They were in the interaction between changed lines and the code those lines touched.
This is not a problem that better prompting solves. You cannot ask a language model to reason about code it has not seen. The fix is architectural: before analysis, you need to build the context that makes the diff meaningful. For each changed function, you need to know: what does it call? What are the signatures and contracts of those callees? Who calls it? What invariants do its callers depend on?
At PRCheck, our analysis pipeline starts with parsing the entire changed file (not just the diff) and building a partial call graph for the context surrounding each change. For functions modified in the PR, we extract the signatures, docstrings, and type annotations of the functions they call, one call level down. We also extract the call sites in the same repository where the modified function is used, to understand what the callers expect from it.
This is more expensive than diff analysis. It requires more computation per PR, and it requires fetching additional file content from the repository beyond the diff. We think the trade-off is obvious: an analysis that misses the most common class of PR-level bugs is not worth the latency savings.
False positives erode trust faster than false negatives
A second failure mode we observed in existing tools was a high false-positive rate. Language models asked to review code tend toward thoroughness. When uncertain whether something is a problem, they comment anyway. On a real PR with 200 lines of diff, this can produce 15-20 comments where 2-3 are actionable and the rest are noise: suggestions to add documentation that already exists elsewhere, questions about edge cases that are already handled, style observations that contradict the team's existing conventions.
The consequence of false-positive-heavy review is predictable. Developers scan the comments, identify the noise, and then stop reading carefully. The 3 actionable comments get the same surface-level treatment as the 17 noise comments. The signal is lost.
We spent a significant amount of time on precision tuning before shipping anything. Our current approach uses structured analysis rather than open-ended generation for most categories. Rather than asking "what is wrong with this code?", we run specific checks: does this function now have a code path that does not return a value? Does this modified query now lack parameterization? Do these changes affect the cyclomatic complexity of a function that was already above threshold? These structured checks have defined pass/fail conditions and produce a comment only when the condition is met. The comment references the specific rule and the specific line, not a general observation.
The result is fewer comments per PR, but comments that developers actually read and act on.
What LLM-based analysis is still good for
We are not making the argument that language model analysis has no place in code review. There are categories where the generative approach genuinely adds value.
Summarizing a PR is one. A brief, accurate description of what a PR changes, including its likely impact on callers, is useful both for reviewers and for changelog generation. Language models do this well because it does not require precision; approximation is fine.
Identifying code that is technically correct but confusing is another. A function that works but is difficult to reason about, names a variable ambiguously, or handles a case in an unexpected way is the kind of issue that benefits from a natural-language observation rather than a structured rule check. Human reviewers do this well too, but automated pre-review surfacing of these cases can prime human reviewers to look for them.
Explaining the likely intent of a complex function is a third case. When a reviewer opens a PR modifying a function they have never seen, a one-paragraph explanation of what the function does in context can save five minutes of tracing through the codebase. This is purely additive.
What LLM analysis is not good at, and what we stopped using it for, is making definitive determinations about correctness, security, or performance. Those require precision that generative models cannot reliably provide without structured grounding.
What we actually built
PRCheck's analysis layer is a combination of structured static analysis (AST-based rules for security and correctness patterns), call-graph-aware context extraction (to give the analysis visibility into the surrounding codebase), and targeted generative summarization (for the categories where language models add value without precision requirements).
The structured checks run on every PR. The call graph extraction runs on the files touched by the PR, one call level deep, to provide context. The generative summarization runs on a bounded subset: PR description, diff summary, and impact analysis on known callers.
The combination produces analysis that is faster than a full SAST scan, more precise than pure LLM review, and scoped to the changes in the PR rather than the entire codebase. We think that scoping matters: a PR-level tool should give feedback about the PR, not produce an audit of the whole repository on every merge.
Building this took longer than it would have to build a wrapper around a language model API. We think that is the right trade-off. A tool that developers trust enough to read every comment is worth more than a tool that is fast to build but noisy enough to ignore.