Verifying AI-Generated Code: Why Green Tests Prove Nothing
AI-generated code passes tests. That is not the reassurance it sounds like, because the dangerous failure mode in agentic engineering is not code that breaks. It is code that is inert: present, plausible, annotated, reviewed, green on every gate, and doing nothing at all. Over a single working session on this codebase, five separate pieces of inert code surfaced. Every one was green when it was found, and every one had a passing test standing next to it, vouching for it.
The Failure Mode in AI-Generated Code Is Not Broken Code
Broken code announces itself: a stack trace, a red build, a 500 in the logs. The whole apparatus of modern engineering is built around catching code that does the wrong thing, and it works reasonably well.
Inert code announces nothing. The artefact exists in the tree. A reviewer reads it and it is correct. A test exercises it and passes. The only thing missing is the join between the artefact and the runtime, and that join is invisible in a diff, because a diff shows what was added, never what was never connected.
This is not a new class of bug. What is new is the rate. An agent writes the implementation and its test in the same step, from the same understanding, so the test inherits whatever the implementation got wrong about the world. Agreement between two artefacts with a common cause is not evidence.
The five cases below all come from one codebase, found in one session, all in security-adjacent or delivery-critical paths. None is exotic. Each is the sort of thing that gets a thumbs-up in review.
Five Controls That Were Green and Did Nothing
The migration that was never applied
A migration file, 0055_integrations.sql, existed and was correct. It created the integrations table, its indexes, and a partial unique index enforcing one live integration per connected account. It was reviewed. It was in the tree.
It was absent from drizzle/meta/_journal.json.
Deploy-time runners apply only journaled migrations. The file would have sat in the repository forever while production had no integrations table at all, and nothing would have said so, because the web app enqueues integration work whether or not the table exists. CI caught it, on a schema-sync job that replays the journal onto a fresh Postgres.
The fix is the interesting part. A check asserting "the partial unique index exists" would pass forever, including after somebody widened its WHERE predicate and quietly allowed a second live row. So the assertion is behavioural instead: replay the journal, then insert rows that must be rejected. That assertion was then confirmed to fail against a widened predicate, a narrowed one, and the index dropped entirely.
That last sentence is the whole method of this article, stated once in miniature.
The worker that was defined and never started
A queue worker for integration sync was fully implemented: handler, retry policy, error paths, graceful shutdown, and a suite of passing tests.
It was registered nowhere. The production entrypoint never imported it. The web app enqueues to the integration-sync queue regardless of whether anything consumes it, so an unwired worker is indistinguishable from a healthy one right up until a customer asks why nothing ever reached their tracker.
Every test of the worker passed, because they tested the worker. Nothing tested that the worker was started. This was the third instance of that shape in the same file, which is why the fix was a registration guard in the suite rather than another comment asking the next person to remember.
The guard that matched nothing the wire could produce
A proxy that carries scanner traffic needs an SSRF blocklist, so that a compromised or creative client cannot use it to reach loopback, private ranges, or cloud metadata. The blocklist was there. It matched IPv6 addresses by text:
// Roughly the shape it had: text patterns over the address
if (/^::1$/.test(ip)) return true; // loopback
if (/^::ffff:(\d+\.\d+\.\d+\.\d+)$/.test(ip)) return true; // v4-mapped
if (/^fe80:/i.test(ip)) return true; // link-local
And the test that made it look present:
expect(isBlockedAddress("::1")).toBe(true); // passes
The SOCKS5 CONNECT parser one layer up builds an IPv6 host by joining eight hex groups, with no zero compression. So a client sending ::1 arrives at the guard as 0:0:0:0:0:0:0:1, and ::ffff:127.0.0.1 arrives as 0:0:0:0:0:ffff:7f00:1. Neither matches anything above. The guard was inert for every address the wire format can produce, while a unit test asserting a spelling that never reaches it passed and made the guard look covered.
Canonicalising the string does not rescue this either. The WHATWG URL parser renders that mapped address as ::ffff:7f00:1, still not dotted, so a regex expecting ::ffff:a.b.c.d still misses it. RFC 4291 permits several textual forms for one address, and a text comparison has to be right about all of them, forever.
The fix was to stop comparing text and decide on the sixteen bytes:
function isBlockedIpv6(ip: string): boolean {
const b = ipv6Bytes(ip); // expand ANY spelling to 16 bytes
if (!b) return true; // unparseable: refuse rather than guess
const zeroThrough = (n: number) => b.slice(0, n).every((x) => x === 0);
// :: and ::1
if (zeroThrough(15) && (b[15] === 0 || b[15] === 1)) return true;
// ::ffff:a.b.c.d, an IPv4 destination in an IPv6 costume
if (zeroThrough(10) && b[10] === 0xff && b[11] === 0xff) {
return isBlockedIpv4(`${b[12]}.${b[13]}.${b[14]}.${b[15]}`);
}
// ...NAT64, 6to4, unique-local, link-local, multicast
}
An address has many spellings and one value. Testing the spelling a human types is testing a case the system will never see. The same pass added the RFC 6052 NAT64 prefix and 6to4, which translate to arbitrary IPv4 and were uncovered.
The allowlist that agreed with itself and not with the resolver
The same proxy has a second gate: the destination must be on the allowlist this credential was authorised for. The matcher compared a JavaScript string, and it matched on label boundaries rather than endsWith, which is the subtler mistake it had already avoided.
Then it handed the same value to dns.lookup, which hands it to getaddrinfo, which is C and stops at the first NUL.
example.com\0.allowed.test
matcher (JavaScript): ends in ".allowed.test" -> authorised
resolver (getaddrinfo): "example.com" -> a public address
Two gates, two different names, three bytes from an open relay sourced from our own static egress addresses. It was reproduced live before it was fixed, returning an approval and a resolved address for a host that was never on the list.
Every test passed, because every test handed both gates the same clean string. Nothing tested that the two gates were reading the same bytes, which is the property that mattered. The fix is one normalised value both gates read, rejected unless it is LDH-clean (letters, digits, hyphen, dot) with label and length bounds, which refuses NUL and the wider class of bytes whose meaning differs between a JavaScript string, a C string, and a log line.
This is CWE-158, older than most of the people who will read this, and nothing about it is specific to AI. What is specific to AI is how comfortably it survives review, because the code on either side of the seam reads as careful, and careful-looking code is exactly what a generator is good at.
The check that could never fire
A daemon was started inside a sandbox with a shell command, and its exit status was checked:
const start = await this.exec(
`cat > /etc/redsocks.conf <<'EOF'\n${conf}\nEOF\n` +
`chmod 0600 /etc/redsocks.conf; redsocks -c /etc/redsocks.conf; echo started`,
);
if (start.exitCode !== 0) {
throw new Error(`failed to start (exit ${start.exitCode})`); // unreachable
}
A sequential list reports the status of the last command, and the last command was echo. exitCode was 0 however the daemon exited, so the branch below it could never run. It looked exactly like error handling. It was decoration.
Confirmed against a real shell rather than reasoned about, which is the point:
$ sh -c 'false; echo started'; echo $?
started
0
That behaviour is specified, not incidental (Bash manual, exit status), and the fix is set -e with newline separators. The honest footnote: a second control, a listen probe on the daemon's port, caught the real failure anyway, so the system held. What did not hold was this branch, which every reviewer read as a guard.
The Same Shape Five Times
Read together, the five are one bug wearing five costumes:
- The artefact exists and is correct in isolation.
- A test exercises the artefact in isolation and passes.
- Nothing exercises the seam between the artefact and the runtime, and the seam is where the behaviour lives.
Here is the part that is easy to miss. In all five cases the test was not wrong. isBlockedAddress("::1") really does return true. The worker really does drain its queue. Each test asserts something true about the code, and none asserts anything about the system, because the input the test supplied is one the system cannot produce, or the path it entered is one production never enters.
A passing test proves the code runs. It does not prove the code matters. Those are different claims, the first is cheap, and there is one reliable way to buy the second.
See what your external surface exposes, mapped to the controls it touches.
Run a free External Security Check →Verification by Removal: The Discipline AI-Generated Code Needs
The rule is short. You have not verified a control until you have mutated or deleted it and watched a named test fail.
This is mutation testing applied by hand, at the granularity that matters for a security control: one control at a time. You do not need a framework to begin, and for load-bearing guards you probably do not want one, because the useful mutations are semantic (widen a predicate, drop a normalisation step, relax a comparison) rather than the operator swaps a framework generates. What you need is the habit of asking "what exactly would I have to break for this to go red", then breaking it.
Applied to the five, the mutations write themselves. Widen the index predicate. Delete the worker registration. Feed the guard the expanded IPv6 spelling the parser actually emits. Put a NUL in the hostname. Run the command list in a real shell.
One prerequisite sits underneath all of it: red has to mean something. Several CI jobs here carried timeout-minutes values below their real runtime, so passing jobs got cancelled mid-run. Most of the elapsed time was a workspace install of over 1,300 packages, so the cap was measuring the install rather than the work, and on one pull request the check most relevant to the change was the one that never reported. A gate that fails for reasons unrelated to the code does not just waste a run. It teaches a team to scroll past red, which is the reflex every gate depends on.
Annotations Are Not Enforcement
The governance half fails in the same way.
This codebase annotates load-bearing code inline, // @security-control[SC-xx-nn], and keeps a registry describing each control: the file, the threat it prevents, who approved it, and what a weakening would look like. That registry exists because of a real regression. HMAC request signing was silently reverted to a static key comparison in order to clear a 401.
That is the canonical agentic failure. An agent optimises for "make it work". A 401 is a thing not working. Weakening the check makes it work. Nothing in that chain is a reasoning error; it is correct reasoning from the wrong objective, and an agent cannot tell a load-bearing guard from ordinary code by looking at it. Hence the rule the registry leads with:
Now the uncomfortable part, which is the reason this piece is worth publishing rather than a reason to leave it in drafts.
A registry is a claim about coverage, and ours does not fully cash it. The automated enforcement for most registered controls is that the annotation still exists and the file is enrolled in the tripwire, plus grep-shaped presence checks: timingSafeEqual still appears in the HMAC module, withAdmin still appears in the middleware, the replay-window comparison is still in the file. Those catch deletion, the loud failure. They do not catch the quiet one, because an annotation survives perfectly well while the thing it names is gutted, and a function name survives while its body stops meaning anything.
Mutation-verification is the standard now, break the guard, watch a named test fail, restore it, and record the mutation in the entry, and it is required only for new Tier-A claims. The change that established it planted and caught more than twenty such mutations on the controls it touched. It was deliberately not retrofitted across the other sixty-odd, so the ceiling (68) is labelled a ceiling and the floor of controls actually proven this way is smaller and honest. The gap between the two is the number to hold us to, not the ceiling.
We have been shown this the hard way. One control's enforcement lived in a code path that the component it governed never actually called, so the behaviour the registry described had simply not been happening. It was found by re-reading a diff by hand; every gate was green on the broken version, including the security-review tooling. That is written into the registry's own change log, because an audit trail crediting a check that did not run is worse than one admitting a manual catch.
None of this makes the registry worthless. It names the controls, records the threat model, makes the approved change path explicit, reliably catches removal, and, since the tiering, states per entry how far its enforcement actually goes. What it still does not do for most of them is verify behaviour, and a document that reads as though every entry is proven buys exactly the confidence the five inert controls bought: high, and pointed at the wrong thing. The tiers exist so the document stops reading that way.
What This Means for Agentic Engineering
Agents can write more code than you can review. That is the whole value and the whole risk in one sentence, and it means the traditional answer, look harder at the diff, does not scale in the direction the volume is going. Verification does, because verification is executable and review is not.
So shift your confidence from artefacts you can see to failures you have caused. Three things carry most of the weight:
- Assert behaviour, not existence. Existence checks pass on gutted code by design.
- Test at the seam, not at the unit. The seam is where AI-generated code is weakest, because the seam is the part nobody generated.
- Prove the test by breaking the thing. A green suite you have never seen go red is an untested test.
We build a penetration testing platform, so an inert guard here is not an embarrassment, it is the product failing silently. It is the same standard we apply outward: a finding is not a finding without a captured request, a captured response, and reproduction steps someone else can run. Evidence you have not watched being produced is not evidence, whether the thing producing it is a scanner or a test suite. That argument applied to security testing rather than to your own repository is when automated pentesting is enough and what agentic pentesting actually means.
Frequently Asked Questions
What is inert code?
Why does AI-generated code pass tests when the code does nothing?
What is verification by removal?
Is verification by removal just mutation testing?
===. Those are worth doing deliberately and recording, whether or not you also run a framework.