The day an auditor knocks, you will not be asked whether your agent is good. You will be asked to produce evidence. Decision logs going back six months. Eval scores per release. The change history of the prompt that was in production at 03:14 on a Tuesday in October. The fundamental rights impact assessment you filed before deploying the credit-scoring agent. Either the artifacts exist or they don't. There is no middle ground, and there is no retroactive remediation.

The Moment the Binder Shows Up

It is 9:47 a.m. on a Wednesday in late September 2026. The Chief Risk Officer forwards a letter from the Bundesnetzagentur — the German market surveillance authority for the EU AI Act — and the subject line is simply "Notification of Conformity Assessment." The letter, signed by an Abteilungsleiter you have never met, requests documentation for the company's "AI-assisted hiring agent referenced in the August 2026 LinkedIn post by your Chief People Officer." You have fourteen days to respond. The response must include, at minimum: identity-traceable decision logs for the prior six months, the eval methodology and most recent eval results, the change history of every model, prompt, tool schema, and platform version that was in production during the period, the lineage of any training or fine-tuning data, and the fundamental rights impact assessment for the agent, since employment-decision agents fall under Annex III.

If you have all of this in a folder, the next two weeks are uncomfortable but manageable. If you don't, the next two weeks are an emergency. The point of this lesson is to make sure the folder exists before the letter arrives.

An audit-ready agent is not an agent that has been audited. It is an agent for which the audit evidence has been produced as a continuous byproduct of operation, indexed, retained, and queryable. The evidence is the artifact. The agent's outputs are secondary.

The five evidence categories regulators actually request

Across EU AI Act conformity assessments, NYC Local Law 144 bias audit requests, Colorado SB 24-205 algorithmic discrimination reviews, SEC comment letters on the 10-K AI risk factor, and DPA investigations under GDPR Article 22, the request patterns converge on five categories. If you can produce these five categories on demand for any agent in your program, you are audit-ready.

1. Decision logs — every agent action, with the principal, the input, the tool calls, the output, and the user-facing consequence. Six months minimum retention for EU AI Act high-risk systems; longer for sector-specific rules (HIPAA, FINRA, GDPR right-to-access).
2. Eval evidence — the eval set composition, the scoring methodology, the results per release, and the regression trends over time. The eval is the proof that the agent was tested before deployment and continuously thereafter.
3. Change history — for every artifact that affects agent behavior: the foundation model version, the system prompt, every tool schema, the platform version, and the configuration. Who changed it, when, why, what the eval delta was, and what approval gate it passed.
4. Training-data lineage — only for fine-tuned models or RAG vector stores. What data went in, how it was sourced, what consent or license covered it, what was filtered out, and how PII was handled.
5. Fundamental Rights Impact Assessment (FRIA) — required by EU AI Act Article 27 for deployers of high-risk systems, which includes all Annex III agents (employment, credit, biometric ID, critical infrastructure). The FRIA is a structured document, not a checkbox.

Decision Logs: The Six-Month Retention Rule

Article 12 of the EU AI Act requires automatic logging of events during the operation of high-risk AI systems, and Article 26(5) explicitly requires deployers to keep the logs generated by the system "for a period appropriate to the intended purpose of the high-risk AI system, of at least six months, unless provided otherwise in applicable Union or national law." Six months is the floor. For sector-specific rules — HIPAA's six-year retention, FINRA Rule 4511's six-year retention for books and records, GDPR's right of access which has no fixed retention but practical implication of years — the floor is higher.

What goes in a decision log entry

Auditors want to reconstruct a single agent decision end-to-end. That requires the entry to be self-contained. The minimum schema:

• run_id — unique identifier, used to correlate across systems.
• timestamp_utc — ISO 8601 with millisecond precision.
• agent_id and agent_version — which agent and which deployed version.
• model_id and model_version — Claude Sonnet 4.5, GPT-5.1, Gemini 2.5 Pro, with exact version string.
• prompt_version_hash — SHA-256 of the system prompt in effect at the time.
• tool_schema_version_hash — SHA-256 of the tool catalog in effect.
• platform_version — Vellum, LangGraph, Mastra, Sema4, OpenAI Assistants — with version.
• principal — the user identity on whose behalf the action was taken (OAuth subject claim, SCIM ID), or the service-account ID if no user is involved.
• tenant_id — for multi-tenant systems.
• input — the user-facing prompt or trigger, redacted of PII per your privacy policy.
• tool_calls — an ordered array of every tool invocation: tool name, arguments, result summary, latency, cost.
• output — the final agent response, redacted for PII as policy dictates.
• consequence — the user-facing or system-facing effect: "drafted email to candidate", "approved loan application", "flagged transaction for review".
• hitl_decision — if applicable: who approved, when, and the rationale.
• cost_usd — per-run cost in dollars.
• region — the inference region (us-east-1, eu-west-1, eu-central-1) — critical for data-residency proof.

Storage architecture that satisfies retention without bankrupting you

For an agent program running 50,000 runs per day, six months of full-fidelity logs at 8 KB per entry is roughly 73 GB. That is trivial in raw storage cost — under $2/month on S3 Standard, under 50 cents on S3 Glacier Instant Retrieval. The expensive part is query: an auditor asking "show me every decision the credit-scoring agent made for applicants in Bavaria between July 12 and July 19" needs to be answerable in minutes, not days.

The pattern that works in production: write hot logs to a structured store (ClickHouse, BigQuery, Snowflake) with retention of 90 days for fast query; tier cold logs to S3 (or equivalent) in Parquet with retention of 7 years for regulatory floor; index both with a metadata catalog (Glue, Unity Catalog) so a query can span hot and cold transparently. Costs at 50,000 runs/day: roughly $180/month for hot store plus $40/month for cold store.

An audit log that takes two weeks to query is the same as no log. Build the query layer before you build the agent, not after.

The PII-redaction trap

Decision logs include user inputs and outputs. User inputs often contain PII. GDPR Article 5(1)(c) requires data minimization. The temptation is to log everything in full fidelity "just in case" — but you will fail a DPA review. The opposite temptation is to redact aggressively — but then you cannot reconstruct decisions for the auditor.

The pattern that satisfies both: tokenize PII at write time. Replace names, emails, account numbers, and addresses with stable tokens (PII_PERSON_a8f3, PII_EMAIL_b91c). Store the token-to-value mapping in a separate, access-controlled vault with shorter retention and more aggressive deletion rules. When an auditor needs to see a real value, you can resolve the token under a documented just-in-time access procedure, with that resolution itself logged. Tools that do this out of the box: Skyflow Data Privacy Vault, Piiano Vault, Tonic Textual. For self-build, Microsoft Presidio plus a tokenization vault is the open-source path.

Eval Evidence: The Proof It Was Tested

An auditor's second question, after "show me the logs," is "show me the evals." They are asking: did you test the system before deploying it, and do you continuously test it as it changes? Without eval evidence, you have a system that someone hoped would work. With eval evidence, you have a system that produced a score before going to production, and that score was tracked release over release.

What the eval evidence packet contains

• Eval set composition document — how many examples, sourced from what distribution, who labeled the expected outputs, what edge cases are explicitly represented (the easy 80%, the long-tail 20%), refresh cadence, version history.
• Scoring methodology — exact-match, LLM-as-judge with rubric, human-graded sample, or composite. If LLM-as-judge: which model, which rubric prompt (version-controlled), how is inter-rater reliability validated against human spot-checks.
• Per-release results — score on the eval set for every deployed version of every artifact (model, prompt, tool, platform). A regression trend chart, with annotations explaining drops.
• Failure analysis — for every release where the eval score dropped or stalled, a documented root cause and remediation action.
• Bias and fairness evaluation — for Annex III agents, score per demographic group, with the group definitions explicit. NYC Local Law 144 requires this for employment decision tools; Colorado SB 24-205 expands it to consequential decisions. The eval must include protected-class slicing.

The eval tooling that produces audit-grade artifacts

LangSmith Evaluations, Vellum Evals, Braintrust, Patronus AI, and Promptfoo each produce evaluation runs that are timestamped, version-pinned, and exportable. The difference between an audit-ready eval workflow and an ad-hoc spreadsheet of scores is the export discipline. Every eval run must produce a signed artifact: the eval set hash, the model version, the prompt hash, the tool hash, the platform version, the score per metric, the per-example pass/fail, and a cryptographic signature of the bundle. Store the bundle alongside the decision logs in the immutable evidence store.

Braintrust's Datasets feature, Patronus AI's evaluation snapshots, and Vellum's release gates all produce versioned, exportable eval reports. A pattern from a Fortune 500 financial-services deployment: every eval run writes a JSON manifest to S3 under s3://audit-evidence/{agent_id}/{release_version}/eval_manifest.json with object-lock retention of 7 years. The manifest is the auditor's primary artifact.

Change History: The Four Axes of Drift

When an agent misbehaves at 03:14 on a Tuesday, the regulator does not care what the agent looked like in the demo. They care what was in production at that exact moment. Change history must answer four questions for every minute of the prior six months: which model was in use, which prompt was in use, which tools were available, and which platform version was running.

Model version

Foundation model providers do not always tell you when they have updated a "stable" version. Anthropic publishes exact model strings like claude-sonnet-4-5-20251201; OpenAI's gpt-5.1 aliases shift; Google's Gemini 2.5 Pro has revision sub-numbers. Pin to fully-qualified versions in production, log the exact string in every decision log entry, and maintain a model-version timeline document that shows when each version was promoted to production, with the eval score at promotion.

Prompt version

System prompts evolve. A 0.5% change in wording can move eval scores 8%. Maintain prompts in git, hash them at deploy time, log the hash with every run, and store the full prompt body in your evidence vault keyed by hash. The "prompt registry" pattern is built into Langfuse Prompts, Vellum Prompts, PromptLayer, and Helicone Prompts. The hash-and-log discipline is what makes the regulator's question — "what prompt was running when you denied my client's loan application on October 14?" — answerable in seconds.

Tool schema version

Tools change. A new tool is added. An existing tool's parameter list expands. An MCP server is added to or removed from the allowlist. Each of these changes the agent's effective capability surface. Maintain the tool catalog in version control, hash the catalog at deploy time, log the hash with every run, and store the full catalog in the evidence vault keyed by hash. For MCP-based architectures, the MCP server allowlist is itself a versioned artifact — adding a server is a regulated change.

Platform version

The agent platform itself — Vellum, LangGraph, Mastra, Sema4, AWS Bedrock AgentCore, Azure AI Foundry Agent Service, Google Vertex AI Agent Builder — has versions, and those versions change behavior. A platform upgrade can change retry semantics, tool-call concurrency, context window management, or guardrail policies without any change to your prompt or tools. Log the platform version with every run. When the platform vendor pushes an upgrade, that is a change event in your change history, and the next eval run is the proof that it didn't break anything.

The change history as a queryable timeline

The four axes should be unifiable into a single query interface. "What was the state of the credit-scoring agent at 2026-10-14T03:14:22Z?" should return: model claude-sonnet-4-5-20251201, prompt hash a8f3b91c..., tool catalog hash 71d4e2f9..., platform Vellum 4.7.2, eval score on the most recent run 94.3%, last deploy approver Jane Park, last deploy ticket CHG-2024-8841. Tools that support this query pattern out of the box: Langfuse Sessions, Helicone Properties, Vellum Releases. For roll-your-own: a Snowflake table joining the run logs, the prompt registry, the tool registry, the model registry, and the platform deployment registry, all keyed by timestamp.

Training-Data Lineage: Only for Fine-Tunes and RAG

If you use a foundation model out of the box, the model provider owns the training-data lineage question. Anthropic, OpenAI, Google, Meta — each maintains its own internal documentation of training data, and the operator-deployer's responsibility is bounded to using the model within its terms of service. But the moment you fine-tune a model or build a RAG vector store, the lineage question becomes yours.

Fine-tuning lineage

For every fine-tune run, the evidence packet includes: the base model and version, the training dataset (full content, hash, source documentation), the labeling methodology (human, synthetic, mixed), the consent or license basis for every record (especially for any record containing personal data — GDPR Article 6 lawful basis), the train/validation split, the hyperparameters, the resulting model artifact hash, the eval score on the held-out set, and the date and approver of promotion to production.

The most common 2026 finding in fine-tune audits: training data scraped from public sources without documenting the license basis. The EU AI Act Article 10 and the EU AI Office's October 2025 guidance both require documentation of "the provenance of the datasets, including the methods of data collection." If your fine-tune data came from a web scrape, the documentation must include the scrape source, the date, the robots.txt status at scrape time, and the legal basis under your jurisdiction's data-protection law.

RAG vector-store lineage

For every vector store powering a RAG agent, the evidence packet includes: the source documents (titles, versions, dates added, dates removed), the chunking strategy (size, overlap, semantic vs. fixed), the embedding model and version, the index type (Pinecone, Weaviate, Qdrant, pgvector), the access-control model (per-document, per-tenant, per-user), and the refresh cadence. The lineage matters because a hallucination or a misquote in agent output may be traceable to a stale document in the RAG store — and the auditor will want to see when it was added, who approved it, and why it was the version the agent retrieved.

FRIA: The Document Regulators Actually Read

EU AI Act Article 27 requires deployers of high-risk AI systems — and certain other categories — to conduct a Fundamental Rights Impact Assessment before first deployment. The Article 27 list of deployer types includes bodies governed by public law and private entities providing public services, as well as deployers using high-risk systems for credit-scoring or insurance risk assessment. Once you are in scope, the FRIA is not optional and it is not a checkbox.

What the FRIA must contain

Article 27(1) specifies the contents:

• A description of the deployer's processes in which the high-risk AI system will be used, in line with its intended purpose.
• A description of the period and frequency in which each high-risk AI system is intended to be used.
• The categories of natural persons and groups likely to be affected by its use in the specific context.
• The specific risks of harm likely to have an impact on those affected categories.
• A description of the implementation of human oversight measures, according to the instructions for use.
• The measures to be taken in the case of materialization of those risks, including arrangements for internal governance and complaint mechanisms.

The FRIA template that actually gets used

The European Centre for Algorithmic Transparency published a model FRIA template in March 2026. The German Datenethikkommission and the Dutch Autoriteit Persoonsgegevens both reference it. A complete FRIA for an employment-decision agent typically runs 18-30 pages. It is signed by the deployer's senior responsible person, dated, and filed with the relevant national market surveillance authority (in Germany, the BNetzA; in France, the CNIL; in Spain, the AESIA). Filed FRIAs are subject to update when the system materially changes — a model upgrade, a prompt rewrite, an expansion to a new use case all trigger FRIA revision.

The FRIA is the document the regulator reads first. Decision logs and evals are the evidence. The FRIA is the argument. Skip the FRIA and you will not get to argue the evidence; you will be in non-compliance from sentence one.

The FRIA review cadence

Article 27(2) requires the deployer to "update the impact assessment if any of the elements listed in paragraph 1 has changed or is no longer up to date." In practice, this means quarterly review at minimum, with an immediate review on any material change. Build the FRIA review into your release calendar — every quarterly release of the agent triggers a FRIA delta review, with the reviewer named and the result (no change, minor update, material update) logged in the change history.

The Evidence Vault Pattern

The five categories — decision logs, eval evidence, change history, training-data lineage, FRIA — must live somewhere. The pattern that satisfies regulators across jurisdictions is the immutable evidence vault: an append-only, cryptographically-signed, access-controlled store with retention policies enforced at the storage layer.

The technical recipe

• Storage: S3 with Object Lock in Compliance mode (or Azure Immutable Blob Storage, or GCS Bucket Lock), retention 7 years for FRIA and eval evidence, retention 6 months hot plus 7 years cold for decision logs.
• Signing: every artifact signed with AWS KMS, Azure Key Vault, or GCP KMS. Hash-chain manifests so tampering is detectable.
• Catalog: AWS Glue, Unity Catalog, or self-built Postgres index mapping artifact IDs to S3 paths.
• Query layer: ClickHouse, Snowflake, or BigQuery for the hot 90 days; Athena or Trino for cold tier.
• Access control: per-role IAM with audit logs on every read. The auditor gets a time-bounded, read-only role with logged queries.
• Export: pre-built export packs per request type — "FRIA evidence pack", "Annex III decision log pack", "fine-tune lineage pack" — each producing a signed zip with manifest.

The dress rehearsal

The best operators run a quarterly evidence-pack drill. Pick a random run from the prior month. Produce the complete evidence pack for that run: the decision log entry, the eval score for the deployed version, the change history of model/prompt/tools/platform at that moment, the lineage of any RAG content retrieved, and the most recent FRIA. Time the production of the pack. If it takes more than four hours, your audit-readiness is not real. Iterate the tooling until the pack comes together in under one hour.

The Three Audit Archetypes (and What Each Asks For)

Not every audit is the same. The evidence vault must be sliceable for three distinct archetypes.

The regulator audit (EU AI Act, NYC Local Law 144, Colorado SB 24-205)

Scope: a specific agent or a specific time window. Asks for the FRIA first, then the decision logs, then the eval methodology, then the bias breakdown by protected class. Wants signed artifacts and a clear chain of custody. The auditor expects to be able to verify that the artifacts are immutable and that the timestamps are not after-the-fact.

The customer audit (SOC 2, ISO 42001, customer security questionnaire)

Scope: the whole program. Asks for the control framework first (policy documents, RACI, training records), then a sample of evidence proving controls are operating effectively, then evidence of remediation when controls fail. The customer typically accepts type-2 SOC reports as proxy evidence, but the largest customers send their own teams.

The plaintiff audit (litigation discovery)

Scope: one or more specific decisions that allegedly harmed an individual or a class. Asks for everything that led to that decision: the input, the agent's reasoning trace, the prompt and tools in effect, the model version, the training data, the FRIA, the eval evidence, and the chain of approvals. The plaintiff's expert witness will probe for gaps; gaps imply spoliation, and spoliation implies adverse inference. The discipline you built for regulators saves you here.

The Eight-Week Build Plan

If you start today with no evidence vault, the rough plan to be audit-ready in eight weeks:

1. Week 1-2: Decision-log schema — define the 15-field schema, instrument your top three agents, write to a hot store (ClickHouse or Snowflake) and a cold store (S3 with object-lock).
2. Week 3: PII tokenization — drop in Skyflow or Piiano, or build with Presidio plus a tokenization vault. Verify that hot-store logs contain only tokens.
3. Week 4: Change history — stand up prompt registry (Langfuse, Vellum, PromptLayer), tool registry (your platform's native if available, else a git-versioned JSON), model registry (a simple table), platform deployment registry (CI/CD output to a table). Wire all four into the decision-log writer so every run logs all four hashes.
4. Week 5: Eval evidence — wire LangSmith / Braintrust / Patronus / Vellum eval runs to write signed manifests to S3 on every run. Add a bias slicing dimension for any Annex III agent.
5. Week 6: Training-data lineage — document every fine-tune and every RAG store. Store the documentation in the evidence vault with hash links to source datasets.
6. Week 7: FRIA — for every Annex III agent, complete a FRIA using the ECAT template. Get senior responsible person sign-off. File where required.
7. Week 8: Drill — run two evidence-pack drills. Time them. Fix what was slow. Document the runbook.

The Numbers That Make This Defensible at the Board

An audit-ready posture has a cost. The cost is much lower than the cost of not having it.

• Storage: $200-500/month for an agent program at 50,000 runs/day with seven-year cold retention.
• Tooling: $2,000-8,000/month for eval platform (Braintrust, Patronus, Vellum, LangSmith) plus observability (Langfuse, Helicone, Arize) plus PII vault (Skyflow, Piiano).
• People: one platform engineer at 30% allocation for ongoing operation; one compliance partner at 10% for FRIA review and audit response.
• Avoided cost: EU AI Act non-compliance penalties under Article 99 are up to €35M or 7% of global turnover for prohibited practices, up to €15M or 3% for other violations, including failure to maintain logs or conduct FRIAs. NYC Local Law 144 penalties are $500 per violation per day. Colorado SB 24-205 enforcement, beginning February 1, 2026, is up to $20,000 per violation. A single unanswered regulator request can cost more than five years of evidence-vault operation.

The board doesn't fund audit-readiness because they love compliance. They fund it because the alternative is a regulator fine, a class action, or a customer mass-exodus driven by a leaked incident. The line item is cheap insurance.

Key Takeaways

• An audit-ready agent produces five evidence categories continuously: decision logs (6 months minimum, 7 years for sector floors), eval evidence per release, change history across model/prompt/tools/platform, training-data lineage for fine-tunes and RAG, and a FRIA for every Annex III agent.
• EU AI Act Article 26(5) sets the six-month log retention floor, binding from August 2, 2026. Article 12 requires automatic event logging. Article 27 requires FRIA for high-risk deployers.
• Decision log entries must include 15 fields covering identity, model/prompt/tools/platform versions, input/output, tool calls, cost, region, and consequence — sufficient to reconstruct any single decision end-to-end.
• PII tokenization at write-time (Skyflow, Piiano, Presidio + vault) satisfies GDPR data minimization while preserving the ability to reconstruct decisions on demand.
• Change history across four axes (model, prompt, tools, platform) must be queryable as a timeline: "what was the state at timestamp T?" returns the four hashes plus the eval score at the deploy that produced them.
• Eval evidence must be signed, immutable, versioned, and include bias slicing for Annex III agents. Tools: LangSmith Evaluations, Vellum Evals, Braintrust, Patronus AI, Promptfoo.
• The evidence vault pattern: S3 with Object Lock in Compliance mode, KMS signing, hash-chained manifests, retention 6 months hot / 7 years cold, with pre-built export packs per audit archetype.
• The eight-week build plan is achievable for a typical mid-sized agent program; the ongoing cost is $2K-10K/month against potential penalty exposure of €35M or 7% of global turnover.
• Run quarterly evidence-pack drills. If you cannot produce the complete pack for a random run in under an hour, your audit-readiness is theoretical.