Author: Marten Türk, Data Scientist 

Sometimes, merchants make mistakes. They either charge the wrong amount, forget to send the customer's order out or if they do send the order out, it turns out the backpack that was ordered has the wrong color. If the merchant is unhelpful with resolving this issue, customers who paid with Wise can raise a dispute with supporting documentation, so we can help resolve it.

Such dispute processes don’t always follow a universal deterministic pattern and disputes raised against the same merchants can often have various outcomes depending on each case. If the customer has submitted a valid dispute with supporting documentation, then Wise can help them get a refund by raising a chargeback through our card scheme partners, who then help facilitate the process with the merchant’s bank.

In 2026, we set a goal to decrease our response times to all customer disputes. With an ever-increasing number of customers, we knew that to meet our goal we needed to do some automation. Our data science team saw an opportunity in this and we mapped out the entire dispute process and found the biggest bottleneck that could not be handled by our current automation efforts - document analysis. 

 Discovering the bottleneck

To find the bottleneck, we mapped out the entire dispute process and measured time spent per step. Human agents spend around 10 minutes per case. The first step - document analysis - was both the most time-consuming and the most critical. Since most disputes require reviewing documents, automating this step became our focus.

What makes a valid document

Instead of looking at how human agents process our selected documents, we decided to ground ourselves to the absolute source of truth - card scheme rules. Based on the chargeback requirements outlined by schemes, we mapped out what information needs to be present and correct on each document to build a strong case for a chargeback. A strong case naturally leads to higher chances of the customer getting their money back. We then used the criteria for a valid document to shape our technical architecture.

System requirements

When designing the system, we established a core principle: LLMs interpret the documents,  our business rules make the decision. 

We intentionally avoided having the LLM make final decisions. Instead, we wanted to build a system that would extract structured information from unstructured images - such as receipt amounts, items, transaction dates. This extracted data would then be fed into business rules. This architectural choice decouples documents understanding from business logic. It allows teams to update their validation criteria for different dispute types on the fly, without needing the data science team to step in and change the prompts every time a policy changes.

For example, in most cases we would expect the receipt amount to match the transaction amount. But if the customer claims they were charged the wrong amount, we would instead be looking for a discrepancy between the transaction amount and the one shown on the receipt.

The full pipeline would then need to take in documents, extract structured data and pass that to our rules where a decision on the validity of the documents can be made. If the documents are invalid, the downstream systems trigger the appropriate next steps in the customer journey. 

Evaluation

With the architectural criteria set, we needed to figure out how we could best meet our system requirements while maintaining high performance.

Since the submitted documents are screenshots or photos, traditional image classification techniques were not enough. We did not only need to identify whether an image looked like a receipt or a screenshot of a chat, we needed to understand the text, entities, date, amounts and the customer intent amongst all of it.

This made the problem a good fit for multimodal LLMs.

To evaluate different approaches, we needed a sandbox environment that mirrored production. In collaboration with the operational teams we created a gold-label dataset, evenly split between receipts and proof of communications.

The Experimentation Phase

One of the core principles that guides us in the Spend Risk Data Science team is that every time we build, we build something that is as simple as possible, but as complex as necessary. To achieve our performance targets or human QA performance, we created the following experimentation plan:

1. Start with one-shot prompts and see how the LLM performs

2. Create a chain of LLM calls, each call solving a piece of the extraction.

3. A simple agentic system analysing documents and another LLM as a judge to validate.

What Didn’t Work

One-shot prompts were appealing in their simplicity, but their performance plateaued below our target. The reason behind this is most likely that when given a large task the model has to review everything all at once. This creates long chains of complex reasoning in a single pass, which can easily lead to mistakes [1].

We also decided against the agentic setup. Creating an agent with an LLM judge would have increased latency, costs and system complexity without clear gains. We found that a multi-step pipeline was ultimately easier to evaluate, monitor and debug for this specific use case.

The Winning Solution: An LLM Chain

We settled on the second approach: a multi-step evaluation pipeline customized to specific document types.

Rather than relying on one huge prompt, we designed a chain on sequential LLM calls. Each step handles a micro-task - analysing the image, extracting or propagating information forward. This approach works well for several reasons:

In the end the system outputs a JSON that's ready for our downstream pipelines. Finally, to avoid vendor lock-in, we separated the LLM calls, prompt templates and provider configuration from the rest of the system architecture. This gave us a clean abstraction layer that allows us to switch between models with minimal changes if needed.

Build model-agnostic solutions

Many LLM application patterns and architectures are model-independent, but the AI landscape moves fast. By building an abstraction layer for NF-SHIELD, we ensured we are not tied to a single vendor and can easily swap in better or cheaper models as technology evolves.

Build from first principles, not past habits

Approaching the problem from first principles helped us build an efficient architecture. Instead of asking “How do agents review this today?”, we asked “What makes a document valid for chargeback?”. By grounding our system in the absolute source of truth rather than mimicking human workflows, we avoided extracting unnecessary data, keeping the architecture simple, explainable and easy to maintain.

[1] Tushar Khot et al. (2023). Decomposed Prompting: A Modular Approach For Solving Complex Tasks.

[2] Denny Zhou et al. (2023). Least-to-Most Prompting Enables Complex Reasoning in Large Language Models.

[3] Nelson F. Liu et al. (2024). Lost in the Middle: How Language Models Use Long Contexts.

[4] Ansh Radhakrishnan and Karina Nguyen et al. (2023). Question Decomposition Improves the Faithfulness of Model-Generated Reasoning.