Companion to "A request log is not evidence". A 30-minute exercise you can run against your own stack this afternoon. Engineering guidance, not legal advice.
The point of the exercise is not to score well. It is to find out, before someone external does, which of these questions your current storage can answer at all.
Setup
Pick a real request from roughly eleven months ago. Not a category of request, not a dashboard, one request. If your retention does not reach back that far, that is the first finding: write it down and pick the oldest request you have.
Now answer the following from stored data only. No inference from deployment history, no asking the engineer who remembers, no reading the current config and assuming it held.
The questions
A passing answer names a version identifier that was recorded at the time of the request. A failing answer looks up today's policy, or reconstructs from a git history and a deployment log.
A passing answer gives categories and counts: two email addresses, one national identifier, zero payment instruments. A failing answer is either a raw stored payload, which is a different problem, or nothing at all.
Allowed, redacted, blocked, or held for a human. A passing answer distinguishes "no sensitive data found" from "redaction did not run".
A passing answer is recorded per request. A failing answer infers it from the API key, unless that mapping is itself versioned and retained.
A passing answer links from the request record to the approval that made that model available. A failing answer is a ticket in another system, with no link in either direction.
A passing answer reads as one chain: model turns, tool calls, and their results in order. A failing answer is a series of unrelated HTTP log lines that a human has to reassemble.
A passing answer rests on an append-only structure or equivalent tamper-evidence. A failing answer is "nobody would" or "access is restricted", which are both assertions rather than demonstrations.
A passing answer sets retention by the life of the decision. Six months is the AI Act floor for high-risk logs under your control, not a target, and sector rules or the nature of the decision can push it much higher.
Scoring
| Answered from stored data | Where you stand |
|---|---|
| 0 to 2 | You have observability. Useful, necessary, and not evidence |
| 3 to 5 | You have partial evidence, usually strong on decisions and weak on policy version and approvals |
| 6 to 7 | You have an evidence layer with a gap worth naming in your next review |
| 8 | Run the test with a hostile reader instead of a friendly one |
What tends to fail, and why
Policy version. Almost everyone fails this one first. It is the cheapest to fix going forward and impossible to fix retroactively, which is the argument for doing it this quarter.
Identity context. Jurisdiction and project have to be propagated through the request path, not inferred at query time. This is usually the largest piece of work, because it touches every service in the chain.
Approvals. The data exists, in a ticket system, in a spreadsheet, in someone's inbox. What is missing is the link from the record to the approval, in a form that survives the ticket system being replaced.
Tamper evidence. Teams tend to argue this one rather than build it. The question is not whether anyone edited the record. It is whether you can show that nobody did.
After the test
Three things worth doing with the result, in order:
- Write down which questions failed, with the date you ran the test. That document is itself useful in a security review, because it shows a team that measures rather than assumes.
- Fix the write path before the backfill. Nothing recorded from today forward can be recovered later; everything already stored is already as good as it will get.
- Set the retention window by the longest-lived decision your systems make, then check it against data protection constraints. If the record holds categories and counts rather than payloads, that conversation gets much easier.
The relevant statutory floor for high-risk AI logs under your control is six months, under Article 19 for providers and Article 26(6) for deployers. Following Regulation (EU) 2026/1744, in force since 27 July 2026, those high-risk obligations apply from 2 December 2027 for stand-alone Annex III systems and 2 August 2028 for AI embedded in regulated products.