Engineering

Detecting logic bugs vs syntax errors: why they need different approaches

8 min read
Abstract visualization of logic vs syntax analysis layers

The tools most teams have set up for automated code quality are overwhelmingly oriented toward one category of problem: code that is structurally or stylistically wrong. ESLint, Pylint, RuboCop, gofmt, the TypeScript compiler in strict mode. These are excellent tools, and they have driven real quality improvements. They are also oriented toward the easy end of the detection problem.

Syntax errors and style violations are detectable because they exist at the surface level of the code. The text has the wrong form. The structure violates a rule. No understanding of what the code is supposed to do is required. Detection is pattern matching on the AST at a single node or local context.

Logic bugs are different in kind, not just degree. They exist not in the code's form but in the gap between what the code does and what it is supposed to do. Detecting them requires understanding intent, which static analysis generally does not have, or understanding execution paths through the code, which requires control flow analysis rather than pattern matching.

What syntax-level tools can and cannot catch

Syntax-level tools operate on the AST with local scope. A rule that flags == instead of === in JavaScript looks at a comparison node and its children. It does not need to know anything about the surrounding function, the call stack that led to this code, or what value the compared variable might have at runtime. The rule is correct independently of context.

This is a strength: local rules are cheap to compute and produce very low false positive rates when written well. A linter rule that fires on double-equals comparison in a JavaScript codebase is almost always right.

The limitation is that syntax-level tools cannot model execution. They see the code as a structure, not as a program that will run with specific inputs. Consider this Python function:

def process_payment(order, user):
    if user.payment_method:
        charge(order.total, user.payment_method)
        order.status = 'charged'
    return order

This is syntactically valid. A linter will not flag it. But if user.payment_method can be a non-null object with an invalid state (expired card, insufficient funds), the function charges before checking validity and sets the order to charged regardless of whether the charge succeeded. That is a logic bug. No local AST rule catches it because the bug lives in the interaction between this function's flow and the semantics of the charge function it calls.

Control flow analysis: the step above pattern matching

The next level of analysis is control flow: building a graph of all possible execution paths through a function or module and checking properties of those paths. A control flow graph (CFG) represents each basic block (a maximal sequence of statements with no branches) as a node, with edges representing possible transitions between blocks.

With a CFG, you can ask questions that are impossible to answer with local AST rules: is there an execution path through this function that returns without setting a required variable? Is there a path through which a resource is allocated but never freed? Is there a path where a null value reaches a dereference point?

This is the analysis that enables a class of findings that linters miss. A classic example: reachability analysis. A function that has a branch where it fails to return a value in a typed language may compile fine in languages with lenient type checking, but the unreachable-return path produces undefined behavior at runtime. Control flow analysis identifies the path; local AST analysis does not.

Python's mypy, Rust's borrow checker, TypeScript's strict null checks, and Java's definite assignment analysis all use some form of control flow analysis. The borrow checker is arguably the most sophisticated: it tracks ownership and lifetime of every value through every possible execution path, rejecting programs where any path violates memory safety. This is control flow analysis extended to a full resource ownership model.

Where logic bugs live that require deeper analysis

Beyond control flow within a single function, the category of logic bugs that causes the most production incidents typically involves interactions across function boundaries. A function that handles an input correctly under one calling convention but breaks under a different one. A function that has an invariant that callers are expected to maintain, where a new caller violates the invariant without knowing it existed.

These bugs require interprocedural analysis: analysis that follows call edges across function boundaries. A simple example: a function parse_date(s) that assumes its input is a non-empty string. If the calling context always ensures that, the code works. If a new call site passes a value that can be empty, the function raises an exception. Static analysis that stops at function boundaries will not catch this; analysis that traces the call chain from the new call site through to parse_date and checks the precondition there will catch it.

This is exactly the class of analysis we built for PRCheck's call graph traversal. When a PR changes a function's interface or modifies its internal precondition assumptions, we trace the affected callers and check whether they still satisfy the postconditions the changed function now expects. It is not complete (dynamic dispatch and cross-repository calls are out of scope), but it covers the common case of direct function calls within a repository.

The false positive trade-off

There is an important counterpoint here. Deeper analysis produces more false positives than surface-level rules. A linter rule for double-equals comparison has near-zero false positives because the rule is unambiguous and context-independent. A control flow analysis that flags "possible null dereference on path X" may be wrong because the calling context guarantees the value is non-null, and the analysis cannot see that guarantee.

This is the fundamental tension in static analysis: coverage versus precision. More coverage means more true positives and more false positives. More precision means fewer false positives but more false negatives. Every static analysis tool makes choices on this axis.

For code review specifically, false positives are expensive: a reviewer who sees three false positive findings on a PR is going to trust the tool less on the fourth finding, which might be real. The false positive rate directly affects whether the tool's findings are acted on. This is why precision matters as much as coverage in a code review context, and why we spent significant effort tuning the interprocedural analysis to avoid flagging paths that are provably safe given known preconditions.

We are not saying linters should be replaced by deeper analysis. The two layers serve different purposes: linters handle the surface-level issues that are fast and cheap to catch, and deeper analysis handles the issues that require execution path understanding. The value of having both is that the linter's precision at the surface level sets a quality floor, and the deeper analysis provides a separate, higher-coverage pass for the issues that matter most.

Practical implications for PR review tooling

If a team's automated review process consists only of linting and type checking, it is catching a real but limited category of issues. The bugs that cause production incidents are disproportionately logic bugs: incorrect business logic, missed error paths, invariant violations, race conditions in async code. These are the bugs that reviewers are supposed to catch, and the tools that developers think of as "automated code quality checks" are mostly not covering this category.

Extending coverage to logic bugs requires control flow analysis at minimum, and interprocedural analysis for the cross-function category. Both are computationally heavier than linting and require language-specific implementation. They also require tuning for false positive rate per codebase, because what constitutes a safe assumption varies by project.

The practical path for most teams is not to build this analysis themselves but to select tools that provide it and configure them to match the codebase's conventions. Understanding what layer of analysis a tool operates at helps evaluate its coverage claims honestly. A tool that is fast and easy to set up is probably operating at the surface layer. A tool with per-language support, configurable interprocedural depth, and non-trivial setup time is probably doing deeper analysis. The depth matters more for the bugs that matter most.

Catch issues before code ships

PRCheck reviews every pull request the moment it opens. Start in two minutes.