#13 - The 4 hidden drivers behind false positives

Fraud False Positives are top of mind for every Fintech leader.

That’s nothing new, and I’ve touched on it in the past from several angles.

But the question remains - if it’s top of mind for all leadership teams, how come this problem is so endemic in the industry?

And so today I want to take a step back and tackle these questions:

Why everyone should care about False Positives, why they happen, and how can we mitigate this issue?

False Positives might be costing you more than fraud itself

Looking at industry reports from the last few years, it seems that fraud False Positives average between 18-50%(!) of blocked payments.

Side note: Why such a wide range? It's not only about your industry, but also which report you look at. For example, LexisNexis' "True Cost of Fraud" report from 2018 cites 18-28%, while their 2024 report cites "up to 50%".

Frankly, I find these figure to be optimistic based on my own experience, but let’s stick to a conservative 25%.

It doesn't end there though, as False Positives obviously impact user experience. Especially if these are new customers or high-value ones.

A relatively outdated (2015), although likely still relevant study from Javelin Strategy found that 32% of customers who were declined on a website will not return to it.

This means that you’re not only losing revenue from payments you block, but you’re also decreasing the LTV of the impacted users.

It’s not surprising then, that many businesses see the financial impact of False Positives as at least as severe as fraud losses.

Notice the lack of hard figures?

That’s exactly the issue. It’s hard to calculate, and so it’s underreported as well.

And yet, reducing False Positives is usually a primary goal – if not THE goal – of most fraud teams.

But in order to mitigate False Positives, we need to understand why they happen in the first place.

Well, there can be many different reasons for it, but I want to name the most common four drivers I usually encounter.

Driver 1: Fire and Forget

We often refer to fraud prevention as a game of chess, where you and your opponent are constantly reacting to one-another.

But what happens when you actually win?

Let’s consider it for a moment: You’ve released a rule that stopped ALL fraud.

Side note: I say rule, but this can be a model, a block list, a review process, etc.

Here’s what fraudsters can do next:

Option 1 - they repent their sins, never to commit fraud again.

Option 2 - they changed their attack to a different angle, one you aren’t aware of yet.

Option 3 - they have given up and moved on to attack someone else.

Here’s the thing:

It doesn’t matter which of these options they choose, what you need to ask yourself now is:

“What is my rule currently blocking?”

The answer: False Positives.

Such an extreme example is unlikely to happen in the real world, but you get the picture.

And of course, if you’re not actively monitoring False Positives and adjusting your controls accordingly, they will accumulate with time.

Solution A: Implement a holistic monitoring approach to measure False Positives across all your controls.

Not sure how to set it up? I got you covered - ​The 5 ways to spot False Positives​.

Solution B: Don’t just monitor, put in place alerts as well.

Looking at decline spikes will either uncover emerging fraud attacks or misbehaving risk controls. Regardless, you want to be on it ASAP.

You don’t need any fancy tools here, just ask your engineering team what they already use.

Driver 2: Non-Incremental Measuring

Every fraud organization that pushes rules to production has several policies and processes to ensure safety and effectiveness.

The main policy would likely have to do with “Minimum Acceptance Level of Performance”:

Which KPIs are you measuring your rules by, and what’s the minimum threshold for them to be candidates for going live?

Side note: Once more, I say rule, but this can be a model, a block list, a review process, etc.

Usually, this policy will aim to keep False Positives below a certain threshold.

But here’s what I see most organizations miss: they measure the performance of the rule on its own.

In reality though, it might very well be that other controls will decline some of the fraud it catches, which in turn raises its False Positive Rate.

Sometimes, I find that it is not the result of lack of awareness. Instead, it’s the result of a poor data environment that makes incremental testing hard/expensive.

Whatever the case may be, it’s killing your business.

Solution C: Don’t compromise on measuring the incremental performance. Measure what would be the effect of the rule on your entire system: other rules, models, block lists, etc.

If you’re lacking the capability to do so - prioritize obtaining it.

Solution D: Don’t just measure your performance before go-live. Make sure rules are measured incrementally on a regular basis.

It can be that using a new fraud model will suddenly make a lot of legacy rules detrimental to your performance.

Make sure to depreciate them to keep False Positives down.

Driver 3: Poor Data Quality

Nothing like poor data quality to ruin the party.

But don’t confuse data quality with data availability. Yes, availability is important but it’s easy to spot and fix.

However, data quality issues might be lurking under the surface and will require manual investigation to uncover and fix.

Some common problems you may want to look for:

• Crossed data-points: How many times have you seen Last Name appearing in First Name and vice versa? Or an address that is parsed into the wrong variables?

• Default values: Internal IPs, Apple Private Relay emails, a Partner's business phone number… You name it, I’ve seen it.

Side note: ​The most famous story​ about default values causing False Positives.

• Corrupted data: whether it’s a misconfigured timestamp, a special character (e.g., Å,Æ, ß, etc.), or a currency that didn’t convert correctly - be sure that these bugs will hamper your performance.

Solution E: Monitor performance, and especially False Positives, by flow.

And by “flow” I don’t mean geography, payment method, or product. Think instead about data flows: data integrations, partners, web vs. mobile, etc.

If one of them starts to misbehave, chances are you’ll uncover a data quality issue as the root cause.

Solution F: Introduce logging, monitoring and alerting of your data quality.

Frankly, this is easier said than done. But if you find recurring data quality issues, it will be worth it in the long run. And it is still easier than monitoring False Positives themselves.

Driver 4: Unoptimized AI Strategy

I’ve written about it in the past, but it must be mentioned again.

I’ve yet to come across an organization that has truly optimized their use of AI. And while they are busy writing rules instead, their AI is responsible for many of their False Positives.

I’m not going to get into the details of why that is the case, and what can be done to solve it.

If you wish to explore it further, ​check out this TSFS issue​.

To Summarize

False Positives are likely as costly to your business as fraud losses are.

And if you’re not liable for fraud, it’s a certainty.

Instead of chasing your own tail, focus on identifying and mitigating the top 4 False Positive drivers:

• Actively monitor all your risk controls at all times

• Verify performance of new controls on top of your entire existing fraud system

• Monitor your data feeds to quickly identify data quality issues

• Optimize your AI strategy

Have questions or feedback? Reply to this email, I read all messages.

That’s all for this week.

See you next Saturday.

P.S. If you feel like you're running out of time and need some expert advice with getting your fraud strategy on track, here's how I can help you:

Fraud Strategy "Power Call" - Book a consultation call with me to get clear, actionable recommendations that fit your budget. Guaranteed. ​
Book a Call Now >>​

Fraud Strategy Workshop - are you an early-stage Fintech that needs to move fast and with confidence? Book this 1.5-hours workshop to get instant insight into your vulnerabilities, optimization opportunities, and get clear actionable recommendations that won't burn through your budget. ​
Book Your Workshop Now >>​

Fraud Strategy Transformation Program - are you a growth-stage Fintech in need for performance optimization or expansion of your products offering? Sign up to this 6-8 weeks program, culminating in a tailored made, high-ROI roadmap that will unlock world-class performance.
​Schedule a Call Now >>

Enjoyed this and want to read more? Sign up to my newsletter to get fresh, practical insights weekly!