What a digital lender’s backtest revealed about fraud its existing stack missed

Most lenders do not have a shortage of controls. Identity verification, KYC, bureau checks and device intelligence each run on every application, and each can return a clean result. Yet some fraudulent applicants still pass through all of them, and only surface months later as losses.

A fintech credit card issuer wanted to measure that gap directly. Rather than review a handful of suspicious cases, it ran a backtest: how much additional risk would broader identity intelligence have surfaced in applications its existing stack had already approved?

The limitation of isolated identity checks

Each control answers a narrow question well. The difficulty is that a well-constructed synthetic or stolen identity can satisfy each question separately, because every individual attribute may be real, valid or simply unverifiable.

Identity verification

Confirms that submitted details match a record or document – not that the applicant is the person those details describe.

KYC checks

Establish that a person meets onboarding requirements, usually from the data the applicant provides.

Credit bureau data

Describes credit history. A thin or newly built file leaves little to assess, and synthetic identities can build one deliberately.

Device intelligence

Assesses the device and session, which a careful fraudster can keep clean.

Submitted application data

Is, by definition, what the applicant chose to provide.

None of these checks is wrong. But none of them, on its own, asks whether the wider identity is coherent: whether the person behind the application has a consistent footprint across sources that a fraudster does not control.

What the backtest evaluated

Individual case reviews show what went wrong on one application. A backtest shows whether a signal separates risk across a whole population, measured against what actually happened.

1. 01Start with approved applicationsHistorical applications that had already passed the issuer’s existing controls.
2. 02Add a broader identity viewHeka assessed each applicant’s external identity footprint across independent sources.
3. 03Compare with what happenedHeka’s results were matched against confirmed downstream fraud and loss outcomes.
4. 04Measure only the incrementThe objective was the additional information beyond the existing stack – not a replacement for it.

The additional identity evidence

Heka assessed five groups of external signals. Individually, each adds context. Cross-validated, they show whether the claimed identity is consistent – or whether it only exists on the application.

Incremental results

Because every application in the backtest had already passed the existing stack, anything Heka surfaced was incremental by definition. On that basis, the evaluation found 48% additional fraud cases beyond those the existing controls had caught.

Translated into losses, the additional detection represented $1.3M in fraud-loss savings and a 4.82× fraud-loss prevention ROI. The second figure matters as much as the first: incremental data only earns a place in the stack if the losses it prevents outweigh what it costs to run.

Customer evaluation result. Performance varies by portfolio and use case.

What the risk-score distribution revealed

Headline detection rates say whether a signal finds fraud. The distribution of outcomes across score bands says whether the score can be trusted to rank it. Here, loss rates stayed broadly flat through the lower bands, then climbed steeply from 60 upwards.

Two things stand out. The top band is not a small tail – it held more applicants than any band except the lowest – so the intervention point applies to meaningful volume. And the gradient is monotonic where it matters: from 60 upwards, each band carried a higher loss rate than the one before. That is what makes a score usable as a threshold rather than just a flag.

What fraud teams can learn

1. 01Passing conventional checks is not the same as low riskEvery application in this backtest had passed. The additional fraud was found among them.
2. 02Risk becomes clearer when sources are cross-validatedSingle attributes can be real and still describe an identity that does not hold together.
3. 03Validate scores against downstream outcomesA score earns trust when its bands separate confirmed losses, not when it simply flags cases.
4. 04Incremental data matters when it moves an intervention pointThe value lies where a score changes a decision – here, before funding.
5. 05Add evidence without replacing the stackThe existing controls stayed in place. The additional identity evidence complemented them.

Questions to ask in your own portfolio

• Which fraud losses came from applicants who passed every onboarding control?
• Which identity attributes were verified independently, and which were only submitted?
• Do your risk signals correlate with confirmed downstream fraud or loss?
• Which cases could be resolved without adding another manual-review step?
• What incremental evidence is missing from your current stack?

Conclusion

A fraud stack can run every check successfully and still miss the identities that cause the most damage. The better test of its effectiveness is not whether each control returns a result, but whether the stack surfaces the identity risk that matters – before funding, at a point where the decision can still change.