Fraud data science rarely fails because the team can’t build a model. It fails because the “model-ready dataset” is a moving target: production systems evolve, definitions drift between services and analytics, and quick fixes pile up until feature pipelines become brittle, slow, and expensive.

At Wise, we tackled that problem by moving data preparation upstream—pushing feature gathering, standardization, and correctness guarantees upstream—so data scientists could iterate quickly without rebuilding fragile transformation logic for every model.

This is the story of what changed, why it worked, and the engineering patterns that made the improvement durable.

When teams are blocked, they build an abstraction layer to “automate it.” That does remove some dependency friction, but it introduces a new long-term cost: a fragile transformation layer that grows faster than anyone expects.

Two things made it especially painful:

• Output shape mismatch: what production emits is not what data science needs.

• Type/precision drift: even values that look identical aren’t necessarily identical once types and casting enter the picture (e.g., 1234 vs 1234.00).

Over time, the transformation layer becomes a magnet for “just unblock us” patches. Those patches tend not to be a business priority to clean up—until they cause incidents.

The growth curve tells the story: ETL logic expanded from hundreds of lines to thousands as requirements accumulated and edge cases multiplied.

As customers interact with the system, their associated feature values are continuously updated to reflect new behaviors, transactions, and events. Because these updates happen over time, the same feature can have significantly different values depending on the exact moment it is queried or evaluated.

For machine learning training, it is essential that each feature value reflects exactly what was known at that specific moment in time — and nothing that became known later. Failing to do so introduces data leakage and can cause models that perform well during training to have low accuracy in production. 

To better understand the problem, let's take a simple example. Consider a feature that represents how many transfers a customer has made. Over time, as the customer engages with the product, the counter will keep increasing.

The requirements that forced a rethink: online prediction vs offline training

Even though the domains are quite similar, we can distinguish two fundamentally different use cases for features — model training and prediction — each with different characteristics:

Trying to satisfy both by repeatedly re-deriving features downstream is where reliability and velocity suffer. The system needs a design that supports both use cases with shared definitions.

Feature foundations, not feature scripts

Moving data preparation upstream wasn’t just “move one pipeline.” It was about making feature data a platform capability with a few key building blocks.

1) Online + Offline feature stores (with point-in-time replication)

• Treat data as a product

• Automate manual steps

• Design with downstream use in mind

• Invest in contracts + monitoring early—before scale makes the cost unavoidable

• Work across ENG <> PRODUCT <> DATA SCIENCE to keep the pipeline aligned with reality