In short
- 1This guide gives compliance leaders a way to evaluate an AI agent: six scored questions, a checklist for the vendor meeting and the six-week plan Cogentic uses for pilots.
- 2Regulators have cited unreviewed alert backlogs, AUSTRAC has issued infringement notices for late SMRs and the GENIUS Act extends Bank Secrecy Act duties to payment stablecoin issuers.
- 3Start with how deeply the agent works inside your stack, then test citations, who decides and submits and whether it can reconstruct a past case.
Chainalysis, TRM, Elliptic, Sumsub and the Travel Rule networks give a compliance team better signals. None of them writes the case, so analysts still assemble evidence by hand across several tools. AI agents address that gap, and this guide is for the people who must decide whether one is fit to use.
It is written for MLROs, BSA officers and heads of compliance, and for the procurement, model-risk and engineering teams who buy alongside them. It applies whether you choose Cogentic, build in-house or stay on point tools. For a definition of what an agent does, see the companion article, What an AI compliance agent does.
Why compliance leaders are evaluating agents now
An unreviewed queue is a finding. In its January 2023 consent order, the New York Department of Financial Services found that Coinbase's compliance system was overwhelmed by the end of 2021, with a backlog of more than 100,000 unreviewed transaction monitoring alerts, exposing the platform to criminal exploitation.1 DFS's August 2022 order against Robinhood Crypto records a backlog of 4,378 alerts as of 26 October 2020.2
Reporting deadlines are short and enforced. AUSTRAC's guidance requires a suspicious matter report within 24 hours of forming a suspicion related to terrorism financing and within three days for any other matter.3 In May 2025 AUSTRAC issued infringement notices of $75,120 to the digital currency exchange Cointree for allegedly failing to submit SMRs on time.4
New regimes add monitoring duties. Under the US GENIUS Act, a permitted payment stablecoin issuer is treated as a financial institution for the purposes of the Bank Secrecy Act. It must run an effective AML programme, monitor and report suspicious transactions and keep the technical capability to block, freeze and reject impermissible transactions.5 The Act takes effect on the earlier of 18 months after enactment on 18 July 2025 and 120 days after final implementing regulations.5
Six questions to put to any vendor
Score each from 1 to 5. A score below 3 on questions 1 to 3 should end the evaluation.
- 1. How deep does it work inside your stack? It should read from your analytics, KYC, screening, custody and case systems without a migration. Ask which capabilities you can keep from your current providers and which you can take from the vendor, one at a time.
- 2. What does each claim cite? Have the vendor draft a report, then pick a sentence and ask which record it came from. A fabricated fact in a filing is a regulatory event.
- 3. Who decides and who submits? A named person on your team must review the evidence, decide and submit. The product must not be able to file on its own. Ask how it keeps the person who prepares a report apart from the person who approves it.
- 4. Can it show a past case in full? Ask for a case from six months ago with its inputs, the procedure version and the decision. Anything less will fail a regulator's request for the record.
- 5. What model-risk documentation comes with it? Ask for model versions, known failure modes and performance evidence, and whether your model-risk team can test it independently.
- 6. Which jurisdictions are live? Ask for the list of regulators and report formats supported today, and which are planned. Start the pilot where coverage is real.
Match the regime to the test
| Segment | Pressure | What to test |
|---|---|---|
| Australian VASPs | SMR deadlines of 24 hours and three days [3] | Time from alert to a reviewed SMR draft, using your own cases |
| Stablecoin issuers (US) | Bank Secrecy Act treatment and the ability to block and freeze [5] | Evidence that monitoring covers redemption flows and that every decision is on the record |
| Stablecoin payment platforms | Travel Rule exchanges with counterparties in several jurisdictions | Counterparty context in the case, whichever protocol the counterparty runs |
| Issuers and platforms in the EU | Requirements that differ by regulator; confirm each directly | The vendor's live jurisdictions and report formats, in writing |
A checklist for the vendor meeting
- Onchain depth. Wallets, customers and counterparties resolved as one record, with attribution from your analytics provider visible in each case.
- Evidence and audit. Each action recorded as it happens, evidence kept as it stood at the time and a full export of a case on request.
- Integrations. Read access to your analytics, KYC, screening, custody and case systems without custom ETL, or the vendor's own capability for any layer you want to hand over.
- Model governance. Documentation up front, independent testing supported and a way to test for claims with no source.
- Data handling. Retention settings, whether your data trains any model and where it is stored. Ask for each answer in writing.
How a Cogentic pilot runs
Adapt this to your vendor and stack.
- Week 1: scope the integration. Agree in writing which sources the agent reads and what success looks like.
- Weeks 2 to 3: calibrate. Encode your SOPs, jurisdiction rules and thresholds. Review recommendations on historical cases and tune against the overrides.
- Weeks 4 to 6: run live. Work production cases. Your team reviews and submits every filing. Track cases per analyst per week and time from alert to submitted report from day one.
Where pilots go wrong
Adoption. Senior analysts hold years of pattern recognition. Present the agent as added capacity for compliance, because leading with headcount invites resistance.
Trust. If reviewers do not trust the draft they rewrite it, and the time saved disappears. Source links and evidence quality earn trust.
Procurement. Compliance, legal, model risk, security and IT each hold a veto. Run the diligence pack, with model documentation, security controls and a sample case file, alongside the pilot.
Cogentic handles
- Reading your tools in place, or supplying a layer on request
- Investigating each case and drafting the SAR or SMR from it
- Recording each source and action against the case
Your team still owns
- Setting the evaluation criteria and the pilot scope
- Reviewing every case file and report
- Deciding, and submitting to the regulator
Frequently asked questions
Who is this guide for?
MLROs, BSA officers and heads of compliance, and the procurement, model-risk and engineering teams who buy alongside them.
Does the guide apply if we build in-house?
Yes. The questions test any agent, whether you buy one, build one or extend point tools.
Why score some questions as mandatory?
Stack integration, citations and who decides and submits decide whether an agent can be used in regulated work at all. A weak score on any of them should end the evaluation.
How long should a pilot run?
A Cogentic pilot runs six weeks: one to scope the integration, two to calibrate on historical cases and three to run live.
Sources
- 1New York State Department of Financial Services, Consent Order: Coinbase, Inc., 4 January 2023
- 2New York State Department of Financial Services, Consent Order: Robinhood Crypto, LLC, 1 August 2022
- 3AUSTRAC, Suspicious Matter Reports (SMRs): fact sheet for remittance service providers, Published March 2025, before the reformed AML/CTF regime.
- 4AUSTRAC, Crypto exchange $75k out of pocket for missing reporting deadlines, 16 May 2025
- 5US Congress, Guiding and Establishing National Innovation for U.S. Stablecoins (GENIUS) Act, Public Law 119-27, sections 4 and 20, 18 July 2025
Written by
Cogentic Team · Compliance Team
