The download includes the before, intentionally buggy and fixed JavaScript versions, their tests, a local verification script and recorded results. The CI workflow is a template; it has not been run in hosted CI.
Download verification examples (ZIP)Define cross-file reasoning before measuring it
A useful test distinguishes three cases. A self-contained defect can be found in one changed function. A defect spanning changed files requires connecting multiple parts of the diff. A defect involving an unchanged consumer requires looking beyond the edited lines. These cases demand different context and should not share an unexplained score.
Also separate business contracts, security data flow and vulnerable dependencies. A scanner may track tainted data across supported files without understanding the business meaning of a price. A dependency scanner may report a known package vulnerability without evaluating any producer-consumer contract. Choose a defect class before choosing the evaluation tools.
The requirement and the intentionally broken migration
Our demonstration contains catalog.mjs and invoice.mjs. The catalog originally returns prices in dollars. The invoice multiplies a unit price by quantity and one hundred to return cents. The independent requirement is simple: two basic items priced at nineteen dollars must total 3,800 cents.
The buggy migration changes only the producer’s values from 19 and 29 to 1900 and 2900 while keeping the public function name getUnitPrice. The consumer is deliberately unchanged. It still multiplies by one hundred, so the result is one hundred times too large. The invoice code in before and buggy has the same SHA-256 hash in the recorded manifest.
This is an example we designed for teaching. It is not extracted from a customer, attributed to an AI author or used to claim any product’s detection rate. Its small size makes the expected relation easy to inspect.
// Before: catalog returns dollars
getUnitPrice("basic") === 19
// Buggy migration: same API name, now returning cents
getUnitPrice("basic") === 1900
// Unchanged consumer still converts dollars to cents
return Math.round(getUnitPrice(sku) * quantity * 100);Reproduce the business failure
Use Node.js 24 and run the same contract suite against each fixture version. No package installation, model call or network service is involved. The buggy command is expected to exit with code 1; the failing result is part of the evidence rather than an installation problem.
In our recorded local run on Node.js v24.16.0, the two amount assertions fail on buggy and the rejection assertions still pass. Before and fixed each pass four tests. The repair uses the explicit function name getUnitPriceCents and removes the consumer’s extra conversion.
| Assertion | Before | Buggy | Fixed |
|---|---|---|---|
| Two basic items = 3,800 cents | Pass | Fail: 380,000 | Pass |
| Three pro items = 8,700 cents | Pass | Fail: 870,000 | Pass |
| Unknown SKU is rejected | Pass | Pass | Pass |
| Negative quantity is rejected | Pass | Pass | Pass |
FIXTURE_VERSION=before node --test examples/cross-file-units/contract.test.mjs
FIXTURE_VERSION=buggy node --test examples/cross-file-units/contract.test.mjs
FIXTURE_VERSION=fixed node --test examples/cross-file-units/contract.test.mjsWhy syntax success cannot settle this case
We also ran node --check on every catalog and invoice module. All six files passed. That confirms they are syntactically acceptable JavaScript to the tested Node version. It does not establish agreement about the unit returned by an exported function.
A stronger project type could make some unit mistakes harder to express, but a plain number does not carry the business meaning “dollars” or “cents.” Static rules may catch other defects in these modules; the demonstrated amount failure comes from executing an independently specified contract across their boundary.
AI Code Verify inspects a pasted snippet locally in your browser and has no cross-file context or test runner. Pasting the producer or consumer alone cannot establish this example’s relationship. Use the tool only for its displayed checks and run the fixture in your own project environment.
Decide what would count as a useful review finding
For this evaluation, a strong finding names the catalog’s changed unit, the unchanged invoice conversion and a concrete wrong total. A report saying “check arithmetic” is a lead, but it has not established the relationship. A comment suggesting a rename without explaining the defect is useful style feedback, not the same evidence as detecting the regression.
Ask the reviewer which callers it inspected and which were unavailable. If access or context limits prevent it from seeing invoice.mjs, record incomplete coverage. Do not score a timeout or skipped file as an ordinary clean result, and do not infer whole-repository coverage from a statement that the repository was indexed.
Keep both source locations and the requirement with the finding. The expected amount makes it possible for another engineer to confirm the issue without trusting our explanation or the reviewer’s confidence.
Include a negative control and review unsupported claims
The fixture deliberately rejects negative quantities before calculating a total. A claim that negative quantities are accepted is refuted by the guard and its passing test. This gives a simple negative control for evaluating whether a review produces plausible but unsupported criticism.
One negative control is not an accuracy estimate. A broader evaluation needs clean changes and several defect types across representative code. Keep unconfirmed claims separate from verified false positives until someone has inspected the relevant context. A surprising finding may expose a real issue that was absent from the initial answer key.
Do not give the reviewer the later fix or a historical review discussion that reveals the intended answer. Evaluate the defective revision with the same available requirements and context for each candidate. Otherwise the experiment may measure answer recovery rather than useful analysis.
Use a worksheet rather than a headline ranking
The following fields make the result auditable. Keep the actual review output and reproduction beside the worksheet. If you later compare tools, record their mode, version or run date, configuration and context limits. The current article has not run any third-party reviewer, so its worksheet contains no vendor scores.
| Field | What to record |
|---|---|
| Defect and revision | Requirement, defective source and inspected commit |
| Relationship found | Producer and unchanged consumer identified, with evidence |
| Reproduction | Input, expected output, actual output and command |
| Finding quality | Confirmed, refuted, unresolved or non-defect advice |
| Coverage | Files inspected, skipped, unsupported or unavailable |
| Operational cost | Latency, reruns, fees and human validation minutes |
Interpret metrics according to their definitions
If you build a larger labelled set, report cross-file recall separately from all-defect recall. Measure finding precision after adjudicating substantive defect claims, and show false comments per change or the time spent investigating them. State the denominator and treatment of duplicates, style suggestions and incomplete runs.
A suggestion that a developer later fixes is not automatically a confirmed defect. A comment not marked wrong has not necessarily been audited. Results from different versions, datasets, prompts and definitions cannot be turned into a fair percentage ranking by placing them in adjacent columns.
Use the demonstration to test an evaluation method first. Then expand with API schema changes, shared configuration or another relationship that matters to your project. Keep the acceptance criterion tied to useful, reproducible findings rather than comment volume.
Our single-snippet static checker runs local fixed rules. It does not connect a repository, execute your tests, perform cross-file analysis or prove a program correct. Use it as one layer of the workflow.
Sources and editorial notes
Product capabilities can change. Links below are primary references used for this guide; examples are demonstrations unless explicitly stated otherwise. Suggest a correction.