Practice · For CTOs, security architects, engineering leaders
How AI can help your existing architecture, and where it can't
An honest account of AI in security architecture review: what it does well (reading everything, connecting evidence, drafting hypotheses), what it cannot do, and how to tell the difference in any tool.
AI can read every file in your estate before lunch. It cannot tell you what it didn't read, and it cannot own the decision.
The honest version
Most organisations already have an architecture; what they lack is a current, shared understanding of it. The knowledge is scattered across repositories, infrastructure code, pipeline definitions, cloud consoles, tickets and the memories of people who have since moved teams. A review starts by reassembling that picture, and that is often where the time goes.
This is where AI earns its place. Language models are good at reading a great deal of heterogeneous material quickly and connecting it. They are also fluent, which is a risk: a fluent answer is not a correct one, and a model rarely signals what it did not see. Used well, AI makes reviews broader and faster. Used carelessly, it produces confident architecture fiction.
Where AI genuinely helps
- Reconstructing the system. Reading code, infrastructure as code, pipeline configuration and cloud metadata to propose components, relationships, identities and data stores, each linked to where it was found.
- Asking every question, everywhere. A reviewer can check authorisation on five endpoints in an afternoon. A well-directed agent can ask the same question of every endpoint, every role and every queue consumer, and return the ones that need a person.
- Drafting threat hypotheses. Given a boundary crossing and its facts, proposing what could go wrong, which is a good starting point for a threat model as long as the output is treated as hypotheses.
- Explaining unfamiliar areas. Summarising how a subsystem works for a newcomer, with links to the code that proves it.
- Spotting drift. Comparing the architecture a team intended with what the evidence shows, and noticing when a change adds an entry point, an identity or a data flow.
Where it can't, and what to do instead
It cannot see what it was not shown. If runtime behaviour, a manual console change or a vendor integration is absent from the evidence, the model cannot know about it, and it will not reliably tell you so. Require every conclusion to cite its evidence, and require explicit coverage gaps where evidence is missing.
It can be confidently wrong. Research on hallucination and on models' difficulty correcting their own reasoning, and studies showing that information buried in the middle of long inputs is used less reliably, all point the same way: plausibility is not verification. Separate generating a claim from checking it, and check against the evidence rather than against the model's own confidence.
It can be steered by what it reads. A repository, a ticket or a web page can contain text written to manipulate a model. Prompt injection is first on OWASP's list of risks for language-model applications for a reason. Treat everything an AI reviewer reads as untrusted input, never let it execute repository code, and never give it the power to approve its own conclusions.
It cannot be accountable. Accepting a risk, merging a change, attesting to a report: these are decisions someone must own. Frameworks such as NIST's AI Risk Management Framework put governance and human oversight at the centre for this reason.
It does not know your context. Which data matters most, what your risk appetite is, which customer commitments constrain a design: these live with your team, and they decide what a finding means.
Separate facts, hypotheses and decisions
The pattern that makes AI-assisted review trustworthy is the same one that makes any review trustworthy: keep what is observed, what is suspected and what is decided apart, and let each layer cite the one below.
Narrow reviewers help. An agent asked to assess one class of attack against one boundary produces claims that are easier to check than an agent asked to "find security issues". And the check should be able to return "nothing found" or "not enough evidence" as legitimate answers, rather than being pushed to produce something.
Read the diagram as text
- Evidence: Code and IaC, Cloud and CI
- AI-assisted analysis: Architecture model, Specialist reviewers, Evidence check
- Human decision: Findings, Reviewer
- Code and IaC → Architecture model: parsed
- Cloud and CI → Architecture model: collected
- Architecture model → Specialist reviewers: context
- Specialist reviewers → Evidence check: claims
- Evidence check → Findings: cited only
- Findings → Reviewer: decide
Questions to ask any AI review tool
Whether you build or buy, these questions separate useful tools from impressive demos:
- What evidence did it read for this conclusion, and can it show me exactly where?
- What does it say when evidence is missing: a gap, or a guess?
- Can content it reads change its instructions, and how is that contained?
- Does it ever run code from the systems it reviews?
- What can it change without a person approving, and what can it never change?
- How are its outputs evaluated, and how often is it wrong in ways that matter?
- Where does my data go, and is it used to train anything?
Start small, measure, then widen
Pick one system your team knows well. Let an AI-assisted review reconstruct it and raise findings, then compare with what the team knows: what it found that you had missed, what it got wrong and what it could not see. That comparison tells you more than any vendor claim, and it shows where human review should concentrate. Expand to systems you know less well once you trust how the tool behaves when it is uncertain.
Sources
- AI Risk Management Framework (AI 100-1)
NIST
- Generative AI Profile (AI 600-1)
NIST
- Top 10 for Large Language Model Applications
OWASP
- ATLAS: Adversarial Threat Landscape for AI Systems
MITRE
- Guidelines for secure AI system development
UK National Cyber Security Centre
- Lost in the Middle: How Language Models Use Long Contexts
Liu et al., 2023
- Large Language Models Cannot Self-Correct Reasoning Yet
Huang et al., 2023