How Duna measures and maintains AI agent quality in compliance


Hande Kahraman
On this page
Banks that use AI agents in compliance need to trust the work those agents do. But 89% of executives across industries are somewhat or very concerned about errors in AI outputs used for compliance activities, reports PwC’s 2025 Global Compliance Survey.
Quality is measured by how closely AI agents and their results match the work of a human analyst. An agent that misses a high-risk customer can leave analysts doing the same manual checks they were meant to replace.
Duna builds those checks in from the design stage, and uses analyst feedback to keep improving agent performance after deployment.
How Duna’s AI agents support compliance decisions
Duna’s agentic client lifecycle management platform keeps AI agents separate from its deterministic policy engine so that agents never make compliance decisions on their own. Agents generate evidence, and the policy engine decides how that evidence is treated. The evidence can include the information that was gathered, the reasoning behind a suggested result, and a confidence score showing how certain the agent is.
A bank might accept a lower confidence score for a low-risk case but require stronger evidence or analyst review for a high-risk customer.
Think about an agent comparing a customer’s address against a document. It might identify two addresses as a match even though there are spelling inconsistencies or different formatting. The agent can suggest that the information is consistent and explain why. The policy engine determines whether the evidence satisfies the bank’s requirements or needs further review.
Five questions that define agent quality
Duna evaluates each agent against five questions:
Alignment: Does the agent perform the task it was assigned?
Data security: Does it access and use information appropriately?
Domain understanding: Does it understand the compliance concepts needed to complete the task correctly?
Reliability: Does it reach the same decisions a human analyst would, and how does it compare with existing automation?
Usefulness: Do analysts trust and use its outputs, or do they still need to perform the work manually?
Building and maintaining agent quality
Design agents around tasks that are easy to test
Duna designs AI agents as orchestrators that use controlled tools to complete their tasks. Breaking complex work into smaller tasks makes each one easier to test and improve.
Two principles keep agents on task:
Controlled environment: Each agent has a specific job and operates in its own sandbox. The platform determines what information and tools it can access, strengthening data security. The platform also checks that the agent’s actions align with the assigned task. If an agent attempts something outside that scope, Duna terminates the agent and tries again.
Limited scope: Asking one agent to complete multiple tasks makes its results harder to trust. Let’s say a bank needs to map a company’s ultimate beneficial owners (UBOs). Rather than asking one agent to trace every layer of ownership, the platform assigns an agent to identify the owners of one company at a time. Deterministic logic coordinates these checks across the different layers to build the complete ownership structure.
Test agents before deployment and in production
An agent might perform well on a few test cases but produce inconsistent results when it goes into production. Duna tests agents against a bank’s compliance cases before and after deployment:
Historical testing: Before deployment, Duna tests agents against a bank’s historical cases and compares their results with analysts’ previous decisions. Engineers can experiment with different prompts and configurations, then review reports to identify errors and see which changes improve performance.
Shadow testing: Once agents are in production, Duna runs agents continuously in parallel with analysts handling live cases. The analysts don’t see the agents’ suggestions, so their judgments remain independent. Comparing the two sets of results helps Duna assess how reliably agents perform in production.
Engineers can also control which cases are included in an experiment. For example, they can hold back a subset of cases or use feature flags to control where a new agent configuration runs. This lets them compare performance before introducing wider changes. Duna works with each bank to determine which experiments it can run and how historical data can be used.
Measure how useful agents are to analysts
An agent might perform well in testing and meet the bank’s policy requirements. But analysts won’t rely on it if they don’t trust its output and continue doing the work manually. An agent might produce the right answer but provide reasoning that’s hard to follow. Or analysts might routinely check an agent’s work themselves before completing a task.
The following signals help Duna assess how useful agents are to analysts and where they need to improve:
Accepted suggestions show where analysts can use the output without making changes.
Corrections and overrides can identify where an analyst disagrees with the agent’s assessment and provide examples of the expected result.
Discarded suggestions point to outputs the analysts didn’t find useful.
The platform’s event-based logging system records what agents do and why. These records allow Duna’s engineers to build key performance indicators (KPIs), monitor performance, and run experiments to improve results.
Duna is rolling out customer-facing analytics that let customers track their own KPIs.
Use analyst feedback to improve performance
Duna uses analysts’ responses to identify where an agent needs improvement. When an analyst reviews a case, an agent can suggest a value to enter and a comment explaining the decision. The platform compares these suggestions with what the analyst enters, using the bank’s policy to determine how closely the results need to match.
Based on any differences, Duna’s engineers run reinforcement learning. For example, an analyst might reach the same conclusion with a shorter explanation or with details the agent missed. That feedback builds a training dataset to refine agent configurations.
The platform’s internal tools can also recommend updated prompts or configurations, or suggest a different approach to the task itself. For example, an agent might be configured to produce a single decision when the work actually involves several separate decisions. Engineers can then reconsider how the task is structured rather than continuing to adjust the prompt.
Engineers review recommendations and can reject or adjust them. As with updates to code or compliance policies, changes to agent configurations require human approval before entering production.
Make agent activity auditable
Banks need to understand how AI agents reached their results and investigate errors. The platform’s event-based logs record agent actions and their reasoning. When a rule or AI configuration is wrong, these logs help compliance teams understand the impact, correct the error, and see how that change affects the rest of the system.
For external audits and supervisory reviews, Duna’s Due Diligence Files (DDF) let compliance teams download an organized record of a customer’s CDD history with one click, rather than searching through detailed logs.
Quality doesn’t end with deployment
Compliance teams need support they can rely on in everyday work. By keeping agents focused on narrow tasks and learning from analysts’ feedback, Duna can improve their reliability and make their results more useful.
Learn how Duna builds quality assurance into its AI agents to support better compliance decisions. Get in touch.
Banks that use AI agents in compliance need to trust the work those agents do. But 89% of executives across industries are somewhat or very concerned about errors in AI outputs used for compliance activities, reports PwC’s 2025 Global Compliance Survey.
Quality is measured by how closely AI agents and their results match the work of a human analyst. An agent that misses a high-risk customer can leave analysts doing the same manual checks they were meant to replace.
Duna builds those checks in from the design stage, and uses analyst feedback to keep improving agent performance after deployment.
How Duna’s AI agents support compliance decisions
Duna’s agentic client lifecycle management platform keeps AI agents separate from its deterministic policy engine so that agents never make compliance decisions on their own. Agents generate evidence, and the policy engine decides how that evidence is treated. The evidence can include the information that was gathered, the reasoning behind a suggested result, and a confidence score showing how certain the agent is.
A bank might accept a lower confidence score for a low-risk case but require stronger evidence or analyst review for a high-risk customer.
Think about an agent comparing a customer’s address against a document. It might identify two addresses as a match even though there are spelling inconsistencies or different formatting. The agent can suggest that the information is consistent and explain why. The policy engine determines whether the evidence satisfies the bank’s requirements or needs further review.
Five questions that define agent quality
Duna evaluates each agent against five questions:
Alignment: Does the agent perform the task it was assigned?
Data security: Does it access and use information appropriately?
Domain understanding: Does it understand the compliance concepts needed to complete the task correctly?
Reliability: Does it reach the same decisions a human analyst would, and how does it compare with existing automation?
Usefulness: Do analysts trust and use its outputs, or do they still need to perform the work manually?
Building and maintaining agent quality
Design agents around tasks that are easy to test
Duna designs AI agents as orchestrators that use controlled tools to complete their tasks. Breaking complex work into smaller tasks makes each one easier to test and improve.
Two principles keep agents on task:
Controlled environment: Each agent has a specific job and operates in its own sandbox. The platform determines what information and tools it can access, strengthening data security. The platform also checks that the agent’s actions align with the assigned task. If an agent attempts something outside that scope, Duna terminates the agent and tries again.
Limited scope: Asking one agent to complete multiple tasks makes its results harder to trust. Let’s say a bank needs to map a company’s ultimate beneficial owners (UBOs). Rather than asking one agent to trace every layer of ownership, the platform assigns an agent to identify the owners of one company at a time. Deterministic logic coordinates these checks across the different layers to build the complete ownership structure.
Test agents before deployment and in production
An agent might perform well on a few test cases but produce inconsistent results when it goes into production. Duna tests agents against a bank’s compliance cases before and after deployment:
Historical testing: Before deployment, Duna tests agents against a bank’s historical cases and compares their results with analysts’ previous decisions. Engineers can experiment with different prompts and configurations, then review reports to identify errors and see which changes improve performance.
Shadow testing: Once agents are in production, Duna runs agents continuously in parallel with analysts handling live cases. The analysts don’t see the agents’ suggestions, so their judgments remain independent. Comparing the two sets of results helps Duna assess how reliably agents perform in production.
Engineers can also control which cases are included in an experiment. For example, they can hold back a subset of cases or use feature flags to control where a new agent configuration runs. This lets them compare performance before introducing wider changes. Duna works with each bank to determine which experiments it can run and how historical data can be used.
Measure how useful agents are to analysts
An agent might perform well in testing and meet the bank’s policy requirements. But analysts won’t rely on it if they don’t trust its output and continue doing the work manually. An agent might produce the right answer but provide reasoning that’s hard to follow. Or analysts might routinely check an agent’s work themselves before completing a task.
The following signals help Duna assess how useful agents are to analysts and where they need to improve:
Accepted suggestions show where analysts can use the output without making changes.
Corrections and overrides can identify where an analyst disagrees with the agent’s assessment and provide examples of the expected result.
Discarded suggestions point to outputs the analysts didn’t find useful.
The platform’s event-based logging system records what agents do and why. These records allow Duna’s engineers to build key performance indicators (KPIs), monitor performance, and run experiments to improve results.
Duna is rolling out customer-facing analytics that let customers track their own KPIs.
Use analyst feedback to improve performance
Duna uses analysts’ responses to identify where an agent needs improvement. When an analyst reviews a case, an agent can suggest a value to enter and a comment explaining the decision. The platform compares these suggestions with what the analyst enters, using the bank’s policy to determine how closely the results need to match.
Based on any differences, Duna’s engineers run reinforcement learning. For example, an analyst might reach the same conclusion with a shorter explanation or with details the agent missed. That feedback builds a training dataset to refine agent configurations.
The platform’s internal tools can also recommend updated prompts or configurations, or suggest a different approach to the task itself. For example, an agent might be configured to produce a single decision when the work actually involves several separate decisions. Engineers can then reconsider how the task is structured rather than continuing to adjust the prompt.
Engineers review recommendations and can reject or adjust them. As with updates to code or compliance policies, changes to agent configurations require human approval before entering production.
Make agent activity auditable
Banks need to understand how AI agents reached their results and investigate errors. The platform’s event-based logs record agent actions and their reasoning. When a rule or AI configuration is wrong, these logs help compliance teams understand the impact, correct the error, and see how that change affects the rest of the system.
For external audits and supervisory reviews, Duna’s Due Diligence Files (DDF) let compliance teams download an organized record of a customer’s CDD history with one click, rather than searching through detailed logs.
Quality doesn’t end with deployment
Compliance teams need support they can rely on in everyday work. By keeping agents focused on narrow tasks and learning from analysts’ feedback, Duna can improve their reliability and make their results more useful.
Learn how Duna builds quality assurance into its AI agents to support better compliance decisions. Get in touch.
Continue reading
Industries
Customers
Company

Industries
Customers
Company

Industries
Customers
Company






