82 agents, one bug: running Claude's security scanner on my own code

Anthropic shipped a security-scanning plugin for Claude Code, so I did the obvious narcissistic thing: I pointed it at the codebase that serves this blog. Same repo, same jar, LinkedIn OAuth, an AI concierge, visitor analytics, a public blog with an HTML sanitiser — a decent spread of untrusted-input surface to chew on. What came back was less interesting for the one bug it found than for the nine it threw away.

Not a linter — a team of adversaries

Most SAST tools are a big regex with opinions. claude-security is structured more like a small security team that doesn't trust itself. A whole-repository scan runs as a pipeline of phases, each one a fan-out of independent agents:

• Inventory — a cartographer partitions the tree into components and has to account for every top-level directory (scanned or explicitly skipped, with a reason). Mine came out as ten components: backend web controllers, auth/config, the concierge, visitor tracking, persistence, the two frontends, the brand kit, CI/CD.

• Threat model — one agent per component builds its own model of what an attacker would go after.

• Research — the heavy phase: a researcher per component × vulnerability category (injection, auth-and-access, crypto, memory-and-secrets…), reading the actual code, not pattern-matching it.

• Sweep — a breadth pass to catch what the component grid missed.

• Panel — the part that matters, below.

• Adversarial — a final red-team pass over the survivors.

The run tallied 82 agents over about 21 minutes, all on Opus. The research phase alone dispatched 41 researchers. That is a lot of parallel reading, and it is deliberately expensive — roughly 3.2M tokens of subagent work for one medium-effort scan. This is not a thing you run on every commit; it's a thing you run when you want a genuine second opinion.

The interesting design: it rejects almost everything

The 41 researchers proposed 10 candidate findings. Then every candidate went in front of a three-lens verification panel — three independent agents, each told to refute the finding, needing a 2-of-3 vote to survive. Here's the vote record:

{
  "F1":  { "true": 3, "false": 0 },   // survived (unanimous)
  "F3":  { "true": 1, "false": 2 },   // rejected
  "F7":  { "true": 1, "false": 2 },   // rejected
  "F2, F4, F5, F6, F8, F9, F10": { "true": 0, "false": 3 }  // rejected, unanimous
}

Nine of ten candidates died in the panel — seven of them unanimously. That rejection rate is the product. Anyone who has run a noisy scanner knows the failure mode: 200 “findings,” you skim the first ten, all of them are theoretical, and you close the tab and never trust it again. A tool that guards your attention — that treats a false positive as costing you more than a missed one — is one you'll actually read. The panel is tuned to be the skeptic so you don't have to be.

What got rejected was reassuring, too: candidates poking at the concierge's deliberately tool-less LLM, the hand-wired OAuth workarounds, the server-side HTML sanitiser, the owner-gating. All the load-bearing security decisions held up under agents actively trying to break them.

Why this beats the SAST I use at work

I run Checkmarx at the day job, and its problem isn't that it's wrong — it's that it's noisy. Taint analysis traces a source to a sink by over-approximation; it can't actually reason about whether the sanitiser in the middle works, whether the path is reachable, or whether the framework already escapes the output. So it flags the flow and hands the judgment call to you, and you spend your afternoon marking findings “not exploitable” one by one. The refutation panel does that triage for you — three agents arguing “this is a parameterised query, not injectable” is exactly the human review SAST offloads. But it's not a replacement: Checkmarx still wins on determinism, recall over known sinks, CI-speed and the compliance paperwork my auditors actually accept. The genuinely exciting version isn't either-or — it's stacking them: let Checkmarx do breadth, then pipe its wall of findings through an adversarial panel as a false-positive filter, and read only what survives.

The one that survived

F1 was not dramatic, and that's the point — it's exactly the kind of thing a human review glides past. My visitor geo-lookup resolved a coarse location for each IP through a third-party API, and the default endpoint was plain HTTP:

private String apiUrl = "http://ip-api.com/json"; // CWE-319

So the one piece of PII the app carefully masks before it hits the database — the visitor's full IP — was being shipped to a third party in cleartext. A passive on-path observer could harvest those IPs, and even forge the country/region the app then stores and emails to me. Low severity (you need a network position), but a real, unnecessary leak of the exact data I claim to protect.

From finding to shipped fix

The plugin's fix job is the same trust-nothing shape as the scan. It clones the repo into a throwaway workspace, a generator writes the patch there, and then an independent chain has to sign off on three claims before a patch file is even written: the change is targeted, introduces no new vulnerability, and doesn't change behaviour beyond the fix. A verifier ran the full suite (117 tests, green); then a fresh researcher attacked the diff, tracing the geo values to their sinks to confirm the patch opened nothing new.

The fix itself needed a small twist. You can't just flip the scheme — ip-api.com's HTTPS is a paid tier — so the patch swaps to a keyless HTTPS provider and adapts the response mapping:

// before
private String apiUrl = "http://ip-api.com/json";
// after
private String apiUrl = "https://ipwho.is";

Private-IP skip preserved, best-effort “any error → empty, never block login” semantics preserved, no API key added. It went through my normal branch → review → PR → merge → deploy flow and was live on this box the same afternoon.

Would I run it again?

Yes — but as what it is. It's slow and token-hungry by design; it complements SAST, dependency scanning and human review rather than replacing them, and because it's non-deterministic, the value compounds if you run it periodically and let coverage accumulate. One honest caveat it flagged on itself: no test in my suite exercises the real geo HTTP path, so the “behaviour unchanged” claim rested on review plus a live-API probe, not a shipped test. It told me that instead of hiding it. That candour — a panel that argues itself down to one finding and then admits what it couldn't prove — is the part I didn't expect from an AI tool, and the part that made me trust the one bug it kept.