Building an audit trail for AI-assisted decisions
On this page
When a supervisor reviews an AI-assisted decision, they expect a record showing how the outcome was reached. But when analysts have to piece together an audit file from a detailed change log, determining what’s most important can be difficult.
Regulators are also becoming more specific about what institutions need to demonstrate. In its draft guidelines on ongoing monitoring, the European Union’s Anti-Money Laundering Authority (AMLA) notes that firms should be able to explain how AI is used and what it produces so supervisors can understand and challenge their use.
For compliance teams, that starts with understanding the components of an effective audit file.
What to include in the audit file
In addition to core information about the business, such as who owns the company and what it does, a complete audit report should cover the following:
AI Model: Provider, family, and exact version used
This information helps establish if the model used was appropriate for the task at the time. For example, if a supervisor sees that an older version of a model was used, they might determine that it wasn’t equipped to handle the task reliably. That could call the original assessment into question and require the bank to revisit affected cases.
Prompt: Instructions given to the model
The prompt describes what AI was asked to investigate and how it was asked to assess the evidence. Take an AI agent that assesses the plausibility of a company address. The agent researches the address and might find that the business is located in a residential area. Because plausibility requires some judgment, the prompt shows a supervisor how the AI was directed to make the assessment.
Evidence: The documents and data AI considered
Duna’s evidence model tracks how a case changes and allows evidence to be dismissed, but not deleted, when it’s no longer relevant. Retaining that evidence is important because it may need to be revisited later. That means the audit file can reflect the latest assessment and still hold historical data without including outdated findings in the current view.
This also limits the scope of remediation when something goes wrong. A bank may discover that a prompt was unreliable only after it’s been used across thousands of cases. Duna retains the AI-generated evidence so teams can identify the affected results, run those evaluations again with the corrected prompt, and determine which cases require remediation.
Provenance: Data source and context
Provenance gives a supervisor the context to understand why particular evidence was used. The same information can appear across a company registry, customer declaration, identity document, or other source, and those sources may not carry the same weight.
This becomes important when sources conflict. A customer might state in an onboarding form that an individual owns 40% of a company, while the company registry shows 30%. Provenance shows where each figure came from, giving the supervisor context for judging the evidence behind the assessment.
Duna keeps provenance attached to individual pieces of evidence. That allows a supervisor to trace a finding back to its source if needed, rather than seeing only the information that ultimately appeared in the assessment.
AI response: What the model returned and why
Not only does a supervisor need to know the response AI returned, but how that response contributed to the assessment. The original AI response should also be retained before the system processes it into a finding or action. Even if AI returns the right result, something could go wrong when the application parses or handles that response.
For example, AI might correctly extract a registration number from an incorporation document, but the application could incorrectly interpret that value and flag it as a mismatch. Keeping the original makes it possible to determine if the error came from the AI or the application logic.
Human judgment: The analyst’s role
It’s important to separate how an analyst and AI contributed to an outcome so a supervisor can see the division of work. This helps establish accountability by showing where human judgment was applied.
For example, the registry might list a company as “Duna BV,” while the customer enters “Duna.” AI can flag the mismatch, but an analyst may recognize that the customer has simply left off the legal form and that both names refer to the same company. The report should show that the analyst dismissed the finding and why.
The same applies when an analyst accepts the AI-generated result. Capturing the acceptance confirms that a person reviewed and agreed with the finding.
Recommendations for creating a useful audit file
Build the record as the work happens
Create the audit record during the case rather than assembling it later. Each action should be tied to the evidence and policy behind the decision so the full history is available when the case is reviewed.
Duna does this through an audit log that records each action as part of the case. Teams can use that history to produce the audit file instead of reconstructing what happened from analyst notes and emails.
Separate case history from the decision record
Analysts at some institutions we spoke with still manually compile audit reports, then upload them back into their case management system.
To avoid rebuilding the report each time, keep the case history separate from the decision record. The decision record should be structured data, with the evidence and outcome for each policy question, and rendered into a readable report when needed. This reduces the manual work of compiling reports while keeping the historical data available to rerun past cases against a new policy later.
Have answers ready when a supervisor asks
A focused audit file gives supervisors the relevant information without burying it in the broader case history. When the data has been captured throughout the compliance process, teams can generate the report whenever it’s needed instead of assembling it manually.
Learn how Duna helps you maintain an audit-ready record of every compliance decision. Get in touch.
When a supervisor reviews an AI-assisted decision, they expect a record showing how the outcome was reached. But when analysts have to piece together an audit file from a detailed change log, determining what’s most important can be difficult.
Regulators are also becoming more specific about what institutions need to demonstrate. In its draft guidelines on ongoing monitoring, the European Union’s Anti-Money Laundering Authority (AMLA) notes that firms should be able to explain how AI is used and what it produces so supervisors can understand and challenge their use.
For compliance teams, that starts with understanding the components of an effective audit file.
What to include in the audit file
In addition to core information about the business, such as who owns the company and what it does, a complete audit report should cover the following:
AI Model: Provider, family, and exact version used
This information helps establish if the model used was appropriate for the task at the time. For example, if a supervisor sees that an older version of a model was used, they might determine that it wasn’t equipped to handle the task reliably. That could call the original assessment into question and require the bank to revisit affected cases.
Prompt: Instructions given to the model
The prompt describes what AI was asked to investigate and how it was asked to assess the evidence. Take an AI agent that assesses the plausibility of a company address. The agent researches the address and might find that the business is located in a residential area. Because plausibility requires some judgment, the prompt shows a supervisor how the AI was directed to make the assessment.
Evidence: The documents and data AI considered
Duna’s evidence model tracks how a case changes and allows evidence to be dismissed, but not deleted, when it’s no longer relevant. Retaining that evidence is important because it may need to be revisited later. That means the audit file can reflect the latest assessment and still hold historical data without including outdated findings in the current view.
This also limits the scope of remediation when something goes wrong. A bank may discover that a prompt was unreliable only after it’s been used across thousands of cases. Duna retains the AI-generated evidence so teams can identify the affected results, run those evaluations again with the corrected prompt, and determine which cases require remediation.
Provenance: Data source and context
Provenance gives a supervisor the context to understand why particular evidence was used. The same information can appear across a company registry, customer declaration, identity document, or other source, and those sources may not carry the same weight.
This becomes important when sources conflict. A customer might state in an onboarding form that an individual owns 40% of a company, while the company registry shows 30%. Provenance shows where each figure came from, giving the supervisor context for judging the evidence behind the assessment.
Duna keeps provenance attached to individual pieces of evidence. That allows a supervisor to trace a finding back to its source if needed, rather than seeing only the information that ultimately appeared in the assessment.
AI response: What the model returned and why
Not only does a supervisor need to know the response AI returned, but how that response contributed to the assessment. The original AI response should also be retained before the system processes it into a finding or action. Even if AI returns the right result, something could go wrong when the application parses or handles that response.
For example, AI might correctly extract a registration number from an incorporation document, but the application could incorrectly interpret that value and flag it as a mismatch. Keeping the original makes it possible to determine if the error came from the AI or the application logic.
Human judgment: The analyst’s role
It’s important to separate how an analyst and AI contributed to an outcome so a supervisor can see the division of work. This helps establish accountability by showing where human judgment was applied.
For example, the registry might list a company as “Duna BV,” while the customer enters “Duna.” AI can flag the mismatch, but an analyst may recognize that the customer has simply left off the legal form and that both names refer to the same company. The report should show that the analyst dismissed the finding and why.
The same applies when an analyst accepts the AI-generated result. Capturing the acceptance confirms that a person reviewed and agreed with the finding.
Recommendations for creating a useful audit file
Build the record as the work happens
Create the audit record during the case rather than assembling it later. Each action should be tied to the evidence and policy behind the decision so the full history is available when the case is reviewed.
Duna does this through an audit log that records each action as part of the case. Teams can use that history to produce the audit file instead of reconstructing what happened from analyst notes and emails.
Separate case history from the decision record
Analysts at some institutions we spoke with still manually compile audit reports, then upload them back into their case management system.
To avoid rebuilding the report each time, keep the case history separate from the decision record. The decision record should be structured data, with the evidence and outcome for each policy question, and rendered into a readable report when needed. This reduces the manual work of compiling reports while keeping the historical data available to rerun past cases against a new policy later.
Have answers ready when a supervisor asks
A focused audit file gives supervisors the relevant information without burying it in the broader case history. When the data has been captured throughout the compliance process, teams can generate the report whenever it’s needed instead of assembling it manually.
Learn how Duna helps you maintain an audit-ready record of every compliance decision. Get in touch.
Continue reading
Industries
Customers
Company
Resources
Industries
Customers
Company
Resources
Industries
Customers
Company
Resources