#01 - Optimize your AI score conversion in 4 steps

Nothing is more frustrating than an expensive AI fraud score that under-performs.

Actually, there is one such thing - knowing you’re leaving money on the table because you didn't optimize it.

Whether you’re getting the score from a fraud vendor or whether it’s based on your internal models, it’s often the case that it’s not optimized to your strategy.

The results are unsurprising: You are losing revenue, you are frustrating your customers, and most likely - incurring more fraud than you’d want.

What bothers me, is that most businesses I speak to are resigned to letting this continue.

The common belief is that optimizing an AI model is very expensive in terms of time and resources, and sometimes outright impossible (i.e., when your vendor doesn't care).

Here is the good news - they’re wrong.

To get better performance, you don’t necessarily need to optimize the model itself, but rather the threshold logic you choose to act upon. The professional term for this would be “Model Strategy”.

It usually looks like this:

IF fraud_score > 85 THEN block payment

Here’s the thing: Optimizing your model strategy doesn’t require engineering resources. It also doesn’t require changes from the vendor side.

And you don’t need a Machine Learning expert to optimize it. You can do it all in Excel.

There’s only one prerequisite you need to tick to know you can optimize it. That is, the ability to set multiple score thresholds, combined with another datapoint.

Here’s an example:

IF fraud_score > 80 AND amount < $100 THEN block payment

IF fraud_score > 70 AND amount > $100 THEN block payment

But how do you know how to optimize your performance according to your unique strategy? Here are 4 simple steps I take, which I guarantee will lead to a performance boost.

Ready? Here we go:

STEP 1: Align on Expected Outcome

In order to optimize anything, you need to first understand how you’ll measure it. In fraud prevention, you usually look at fraud rates (%) and approval rates (%).

The terms may change between businesses and use-cases, but we keep getting back to these two core KPIs.

As most Fintechs are regulated and need to maintain a certain average fraud rate, most businesses optimize their thresholds so fraud rates remain just under their red line. From that baseline, they will then try to maximize their potential approval rates.

Knowing which fraud rate you optimize for, and which red lines you don’t want to cross is the first step in boosting your performance.

STEP 2: Plot the Fraud Score ROC Curve

Don’t panic! This is simpler than it sounds.

In order to visualize the current performance, we want to plot it on a simple ​ROC Curve​. Here’s how you do it:

• For each score threshold (let’s say 1-100), you plot a point on the chart which will represent the performance if you block events from this threshold and above.

• For your Y axis, we’ll plot how much fraud you’d be able to catch if you use this threshold. This is also referred to as True Positive Rate

• For your X axis, we’ll plot how many events you’ll end up blocking by choosing this threshold. This is also referred to as False Positive Rate

It should look something like this:

Bonus tip: Once you’ve visualized your current performance, you might realize that even your single basic threshold is not optimized. For example, you might find out that lowering it by 5 points would reduce your block rates but will not increase your fraud rates.

STEP 3: Identify Under-Performing Segments

Now comes the magic part.

Your user-base is not a single cohesive segment with the same performance. Within it, there are different segments that perform differently.

Identifying these, and setting specific thresholds for them, will increase your overall performance.

Let’s take the example we’ve used above. We replace the overall performance graph with two lines, each representing a different segment.

We repeat the first action from step 2 and we just add the amount threshold as well to the score threshold.

It should look something like this:

We can now see that we can optimize both segments by changing their individual thresholds.

Which leaves the million dollar question:

How do you find the segments that might be in need of optimization?

There are two ways:

• By different risk profiles: Segment by country/region, amount bands, products, inter/cross-border, payment method, payment types (subscriptions vs. regular), etc.

• By data quality: Segment by user-flow, mobile vs. web, partner/integration type, etc.

You probably have a good hunch already as to your “risky” and “safe” segments, so just follow your instincts.

STEP 4: Choose New Thresholds, Test, and Implement

Once you’ve identified the best segments to optimize, you’re in the final stretch: Choose the thresholds that will optimize each segment according to the KPIs you’ve defined in step 1.

Next, implement the new Model Strategy within your testing sandbox and make sure you get the same results in an environment as close to production as possible.

And if that went well, implement it in your live environment and safely roll-out the changes. My recommendation would be to first validate it in shadow mode before you go all-in.

To Wrap Up:

If you follow the above steps, you’ll most likely see a considerable improvement in your overall performance without investing much time or resources into it.

And if it went smoothly, think about the possibilities: having multiple buckets per segment, cross-cutting segments, etc. The optimization potential is endless.

Had amazing results? Do share!

Have questions or feedback - reply to this email, I read all messages.

That’s all for this week.

See you next Saturday.

P.S. If you feel like you're running out of time and need some expert advice with getting your fraud strategy on track, here's how I can help you:

Fraud Strategy Workshop - are you an early-stage Fintech that needs to move fast and with confidence? Book this 1.5-hours workshop to get instant insight into your vulnerabilities, optimization opportunities, and get clear actionable recommendations that won't burn through your budget.

​Book Your Workshop Now >>​

Fraud Strategy Transformation Program - are you a growth-stage Fintech in need for performance optimization or expansion of your products offering? Sign up to this 6-8 weeks program, culminating in a tailored made, high-ROI roadmap that will unlock world-class performance.

​Schedule a Call Now >>

Enjoyed this and want to read more? Sign up to my newsletter to get fresh, practical insights weekly!

Keep me posted!

Built with Kit