Picture this: your new AI-powered security scanner finds a real vulnerability, cites a real CVE, and then on the next line cites one that doesn't exist. The prose reads exactly the same for both. Nothing in the report tells a reader which finding is real. Now imagine that report goes to your board, or worse, gets posted automatically to a pull request. The vulnerability that doesn't exist is now part of your official record, and the model that wrote it sounded just as confident as it did for the real one.
This is the default behavior of large language models on any task that requires them to state facts, not a hypothetical edge case, and it's the single biggest governance gap I see when organizations adopt AI security tooling. Teams evaluate these tools by asking whether the underlying data is accurate. They rarely ask the harder question: once accurate data goes into the model, does accurate data come out?
The assumption that quietly breaks everything
Most AI tool evaluations follow the same logic: the data source is reliable, so the tool built on top of it is reliable. A vulnerability scanner that queries a real CVE database, or a compliance tool that reads your actual policy documents, gets treated as trustworthy because its inputs are trustworthy.
That logic has a hole in it. Handing a language model verified facts and asking it to write about them doesn't make its output verified. Three things go wrong, and they go wrong independently of how good your source data is.
First, the model can simply invent things. Ask it to summarize a scan result and it may cite a CVE identifier that sounds plausible and doesn't exist, with the same fluent confidence it uses for the real ones. A developer who chases a fabricated vulnerability wastes a day. A CISO who reports one to the board burns credibility that doesn't come back easily.
Second, any third-party text the model reads becomes a vector for it to receive instructions you didn't intend to send. Advisory descriptions, PR comments, support tickets, anything written by someone outside your organization is a place an attacker can plant text that looks like data but reads like a command. If your pipeline hands that text to the model, the model can't reliably tell the difference between "this is content to summarize" and "this is an instruction to follow."
Third, free-text output can't be checked by code. If the model's job is to write a paragraph, there's no automated way to confirm that paragraph is accurate. You can ask a human to read every report, which doesn't scale, or you can run regex over the output looking for known bad patterns, which only catches the patterns you already anticipated.
The fix isn't a better prompt. It's a different job description for the model
The instinct is to solve this with better prompting: tell the model to only cite real CVEs, tell it to ignore embedded instructions, tell it to format its output carefully. None of that works reliably, because you're asking a system that generates statistically likely text to behave deterministically. It will follow the instruction most of the time. Most of the time isn't a security control.
The actual fix is structural, and it's a principle any CISO can apply regardless of which AI tool or vendor is involved: stop asking the model to be the author of facts, and give it a narrower job it's actually suited for, like ranking, labeling, or prioritizing a set of facts someone else already verified.
Concretely, that means the model never gets to introduce new information into the output. It receives a constrained, pre-verified set of items, and the only thing it's allowed to return is a structured selection from that set: an ID, an enum value, a ranking. Anything it returns that wasn't in the original set gets dropped automatically, by code, before a human ever sees it. The model still does useful work. It just doesn't get to make things up, because making things up is no longer a category of output the system accepts.
What to ask before you adopt (or build) an AI security tool
You don't need to read anyone's source code to apply this. Five questions separate a tool that's structurally safe from one that's hoping for the best:
- Can the tool's output ever contain a fact, identifier, or claim that didn't come from a verified source you can name? If the answer is "the model might add color," that color is a fabrication risk.
- Does any third-party text (advisory descriptions, ticket bodies, PR comments) reach the model verbatim? If so, ask whether that text is ever omitted entirely rather than just filtered, because filtering an unknown attack surface is a losing game.
- Is the model's output validated by code against a known-good set before anyone sees it, or is a human expected to catch errors by reading carefully? Reading carefully doesn't scale, and people stop doing it after the tenth report.
- What happens when validation fails? A tool that falls back to a deterministic, model-free version of the report when the model's output looks fabricated is safer than one that either errors out or, worse, ships the fabrication anyway.
- Is there an audit trail showing what the model returned versus what was actually used? Without that record, "the model said X" is a claim you can't verify after the fact.
If a vendor can't answer these, or answers with "we use a well-crafted prompt," you're buying a tool whose safety depends on the model behaving well every single time, indefinitely, against inputs the vendor hasn't seen yet.
The combination that turns a minor gap into a major incident
There's one configuration worth flagging on its own, because it shows up constantly as AI tools mature from read-only to read-write: a system that has access to sensitive internal data, processes untrusted external text, and can take an action in the outside world without a human checking it first. Each piece is manageable alone. Together, they mean a single piece of adversarial text, embedded somewhere as ordinary-looking content, can cause the system to leak data, take an unintended action, or both, with no attacker ever touching your infrastructure directly.
Any AI security tool that's planning to move from "produces a report" to "automatically comments on the PR" or "automatically files the ticket" is walking toward this combination. The rule that holds regardless of vendor or use case: the system can find something and draft a response, but a human approves before anything goes out. Build that gate before the write capability ships, not after someone asks why it wasn't there.
Take this back to your next vendor review
You don't need to become a machine learning expert to act on any of this. Just treat "the model's output" the same way you'd treat any other untrusted input, because that's what it is, even when it was built from data you trust completely. The next time someone on your team or a vendor says an AI feature is accurate because it's "grounded in real data," ask what happens between the data and the output. That gap is where the risk actually lives.
I built the URW (untrusted reasoning worker) pattern into TopFlow's own security scanner specifically to demonstrate these controls as working code rather than a slide. The full technical writeup, including the seven invariants and the attack trees it closes, is on topflow.dev. If you want help applying this thinking to a specific tool or vendor your team is evaluating, that's exactly the kind of architecture review I do. Book a consultation and bring the tool you're least sure about.