Authors: Volodymyr Panov, Data Scientist and Kristjan Erik Liive, Data Scientist

To understand where LLMs fit into a compliance operations framework, it helps to review the mechanics of modern financial crime. Consider a classic fraud scenario: a scam actor manipulates a victim to pay them £500 for non-existent concert tickets in London. The illicit funds must then be laundered through the financial system via a traditional three-stage process:

At Wise we screen every transfer, the typical operational workflow starts when a user initiates a transfer. Machine learning models score the transfer and user attributes. If an anomaly is detected, an alert creates a manual investigation case to study the suspicion. If the risk is confirmed, an external regulatory report is filed.

 LLM Applications in Compliance Operations

Examples of red flags:

• Pass-Through Activity: Funds are routed through an intermediate account, entering and exiting in nearly identical amounts to act purely as a temporary bridge that obscures the link between the original sender and ultimate recipient.

• High Velocity: Funds received into an account are moved out almost immediately, characterized by rapid turnover, concentrated bursts of speed, or fast multi-party transfers regardless of total volume.

• Structuring: A single large sum is intentionally split into smaller, sequential transactions specifically designed to stay below monitoring or regulatory reporting thresholds.

When our LLM detects these red flags, it surfaces them to the human investigator, who confirms the finding or marks it as a false positive. This continuous loop generates real-time feedback that allows us to continuously optimize our underlying data structures and prompts.

Engineering Realities and Practical Takeaways

Transitioning LLM-powered systems from experimental scripts to a production-grade compliance engine revealed several crucial engineering and operational lessons:

Start Simple and Avoid Over-Engineering

Data scientists, being technical enthusiasts at heart, often have an urge to implement highly complex, cutting-edge architectures right out of the gate. When we first tackled automated report writing, we built a multi-layered multi-agent system. It performed poorly, ran slowly, introduced excessive token costs, and was incredibly difficult to debug. Errors compound so we quickly learned that a simplified, well-structured LLM pipeline is far superior to an over-engineered multi-agent framework.

Human Judgment is Not a Perfect Ground Truth

While human-in-the-loop validation is vital, human judgment can introduce its own complications. When reviewing automated SAR drafts, investigators frequently make minor, stylistic changes based on personal writing style rather than only necessary factual corrections. This introduces unnecessary noise to the optimisation objective. Additionally, when regulatory compliance guidelines change, human teams require significant retraining and adjustment periods. Automated systems can adapt to new prompts or guidelines near-instantly. By tracking agent edit rates and differentiating between minor synonym adjustments and high-importance factual corrections, we keep text-generation aligned.

Prioritize a "Golden Data Set" Early

A golden data set is a human-reviewed, high-quality collection of sample cases that serves as an optimization ground truth. In financial operations, a true ground truth dataset is rarely available out of the box because historical event logs and outcomes are often incomplete or split across disparate data silos. Hand-picking cases manually is slow and rarely yields more than a few hundred samples. The most effective strategy is to gather such data directly from production by capturing the workflows of your most experienced, highly qualified teams as they complete tasks in real-time.

Fixing Gaps or Finding Rules? The Optimizer Dilemma

Model prompting serves to align a model's outcome conditional probability distribution with some business objective. These target outcomes are typically defined by a singular performance metric or qualitative rubric. Prioritizing the refinement of prompts and evaluation frameworks is the most efficient strategy for organizations to enhance performance before investing in the infrastructure required for complex fine-tuning.

Consider a standard text classification objective. While basic prompting provides a sufficient heuristic to cover most common clean-cut samples, you will inevitably reach a plateau where manual prompt iterations yield only marginal gains. This is precisely where frameworks like DSPy GEPA shine. By employing a "teacher" LLM, these systems utilize scalar metrics and qualitative business feedback to evolve prompt candidates based on the underlying “student” model's performance on a holdout set. Integrating automated optimization allows bypassing the most repetitive prompt development struggles, such as initial drafting and routine adjustments.

Conclusion & Takeaways

Deploying production-grade LLM applications within a regulated framework is a continuous development loop that requires: