#04 - The 5 ways to spot False Positives

Here’s a short summary depicting the evolution of anyone who’s ever operated a fraud prevention system:

“I figured out how to stop fraud. I’m invincible!”

“But wait, I think I’m wrongly blocking good users…”

“OMG, I am blocking good users! OK, I’ll just investigate my false positives and tweak my defenses accordingly.”

“Hold on a sec. How do I know which event is actually a false positive?”

And just like that, we all find ourselves struggling with the same challenge.

If you blocked an event, how do you know whether it would've ended up in a fraud complaint? And if you cannot tell that, how would you make better decisions in the future?

Well, the bad news is that you can’t identify 100% of your false positives.

But the good news is – you guessed it – that you can identify enough of them to enable you to improve your performance.

In fact, there are 5 methods you can use for that.

Picking the right one will depend on your situation, but ideally you’re able to implement at least 2 of these for best results.

Logic Simulations

This is where most organizations start, as it’s the easiest to implement.

You simply run your logic on a historical dataset, and check how many good events that previously passed your system will now end up as blocked.

It sounds like an easy solution, but it actually has a few drawbacks: First and foremost, it’s only applicable for automated decisions (unless you found a way to simulate your employees…).

It’s also hard to account for differences between your sandbox and your production environments.

Small discrepancies can lead to inaccurate results easily.

But most importantly, running an historical simulation takes your pre-existing fraud system as the default state. The false positives that are already baked into it will go unnoticed.

Manual Review

You can periodically run a sample of your declined transactions through your case investigations team. Or lacking that, even your fraud analytics team.

The point will not be to tag each and every false positive of course. Instead, you’re aiming to do two things:

The first, get a ballpark KPI figure that tells you where you are. The second, by investigating these events manually, you’ll be able to collect leads for logic optimization.

You can choose to review just a random sample of all blocked events, or you can choose a specific rule/model/workflow to assess.

This works especially well when looking at “fresh” events, as there’s little else you can do.

Of course, it’s a very resource-intensive process, so pick when and where you use it.

Linking

Did you see good events from the same account before or after you blocked it? Can you link these events/sessions to the one you blocked?

In addition, by using “strong” linking logics, you can associate events even across accounts.

For example: a signup from the same IP and physical addresses as an old established account.

Side note: figure out which linking logics are strong and “safe” enough for you. The point is not to have 100% assurance (you can’t). It’s about getting a ballpark of where you are.

Linking new events to mature ones (good or bad users), can be highly effective in highlighting optimization potential. But it does require a bit of investment to run properly.

At scale, and especially if you don’t want to waste a day every time you run it, you’ll probably want to build a small service that will enable you to run it as much as you want.

Control Group

If you have enough traffic, you can implement a small (1%-2%) control group. This segment will get a free “get out of jail card” and will pass through ALL of your fraud defenses.

Before you jump the gun: yes, the assignment to a control group needs to be random and shouldn’t “stick” to specific users, devices, accounts or whatnot.

This can end badly.

But once such a control group is set-up, it’ll give you a small “window” through which you can measure your traffic in a laboratory-like environment.

Just don’t forget to budget for the extra losses.

Asking the users

This option is only available for Fintechs that own the relationship with the user, but it’s an effective one.

Just ask them!

Blocked an event? Send a “was it you?” message to a confirmed device and collect feedback.

Once more, this won’t produce 100% of the results you’re looking for, but it will complement any other method you might be using.

Conclusions

Look, identifying false positives is one of the biggest challenges in fraud prevention.

Everyone is struggling with it.

Everyone.

There are no straightforward solutions, and it’s likely that you’ll need to combine several of the methods I’ve listed above.

But going the extra mile with this one is what separates the good from the great:

Embracing ambiguity. Making decisions with little data. Iterating on experiments. Obsessing over user experience.

Because that's what it takes.

That’s all for this week.

See you next Saturday.

P.S. If you feel like you're running out of time and need some expert advice with getting your fraud strategy on track, here's how I can help you:

Fraud Strategy Workshop - are you an early-stage Fintech that needs to move fast and with confidence? Book this 1.5-hours workshop to get instant insight into your vulnerabilities, optimization opportunities, and get clear actionable recommendations that won't burn through your budget.

​Book Your Workshop Now >>​

Fraud Strategy Transformation Program - are you a growth-stage Fintech in need for performance optimization or expansion of your products offering? Sign up to this 6-8 weeks program, culminating in a tailored made, high-ROI roadmap that will unlock world-class performance.

​Schedule a Call Now >>

Enjoyed this and want to read more? Sign up to my newsletter to get fresh, practical insights weekly!