#14 - Is your fraud team actually accountable?

One of my recent LinkedIn posts touched on the topic of fraud KPIs.

Like most of my posts, I barely got to scratch the surface, but I did get an interesting question from Ches Jones:

“How should a fraud department be structured when it comes to taking accountability for fraud losses?”

(Slight adjustments for clarity)

Frankly, I’ve seen a lot of fraud departments that got it wrong.

That included myself as well, before I realized what the most effective management framework was.

So today I want to talk about what hidden pitfalls you should be aware of, and what I’ve done to avoid them.

How fraud departments are usually set up:

Let’s start with the obvious disclaimer - there are as many different fraud department structures as there are different fraud departments out there.

Sometimes, not all the key functions that influence fraud performance even report to the fraud department.

But I want to identify three key components that mature organizations usually establish to manage fraud:

Fraud Operations Team: We tend to think of FraudOps teams as large organizations made of investigators and operators.

But I look at a wider definition of operations, one that isn’t fixated on case-by-case management.

For example, it might be a team of rule writers that is busy implementing changes daily: rule monitoring, flash fraud mitigation, onboarding of new flows, etc.

I will consider this team as part of the Fraud Operations function.

Feature Engineering Team: This team is responsible for developing the data features that are available to the department.

These features would be available to all other functions and use-cases, whether for manual review, rule writing, or model training.

Modeling Team: This team is in charge of researching, training, and deploying (Machine Learning) models that score events for risk of fraud.

These three teams are the main functions that drive fraud performance.

Again, not all of them would necessarily exist in every company. And if they exist, they won’t necessarily be all part of the fraud department.

And of course, there are other functions you can find in a fraud org as well.

But let’s focus on these three teams and talk about how to introduce a KPI-driven system that encourages employee-level accountability.

Ownership Structure: Choosing Between Two Bad Options

We’ve established our department, and now we want to manage it in a KPI-driven manner. After all, the team influences the company’s KPIs directly.

Here’s the main issue from my experience:

Fraud performance is influenced by all of the above teams. However, I usually see accountability frameworks falling into one of two bad options:

1. The Fraud Operations team is the main accountable party, even though they own only one piece of it.

2. The entire fraud department is the main accountable party, which puts the responsibility on its leader alone.

I hope the issue with option 1 is clear enough. It also creates a situation where the other teams are measured by project completion, without ensuring their actual influence on performance.

But you might ask, what’s the problem with option number 2? Isn’t that the reason the leader is getting paid?

Here are the implications:

If you're the only one on the hook, it's harder to delegate critical tasks to other team members, which creates a decision-making bottleneck. This eventually slows down the team's reaction time.

In parallel, it takes accountability away from most of the team members, which makes it harder for them to see the value and impact of their individual work.

For years I’ve lived in flux, transitioning between these two options, and sometimes even combining them.

It burned me out, it left my managers frustrated, and it numbed the majority of my team.

It took me a few years, and much deliberation with my team, to find a better way of doing things.

Breaking Down the KPI Tree

True accountability begins when it’s measured with KPIs and not with project management tools.

It is exactly like telling employees what they need to achieve, instead of how they need to achieve it.

The key for distributing ownership is the department leader’s capability to break-down their own KPIs, and streamline them downwards to team leads and team members.

Let’s look at it in reverse order, as that will make the flow clearer:

Modeling Team: Measure incremental model improvements.

How you measure your models’ performance matters less, but I would recommend starting with a simple ​AUC​ tracking.

A short explanation:

Model performance is commonly represented as a curved graph line (ROC curve), showing True Positive Rates and False Positive Rates.

AUC stands for “Area Under the Curve”, and refers to the size of the area that is below that line.

Here’s an example:

Side note: A bit more advanced, is to measure AUC only in the relevant performance range. For example, if your overall block rate is 5%, focusing on the AUC up to 10% FPR makes this metric much more relevant.

Now let’s say your current AUC is 0.8 and the team’s goal is to increase it to 0.85 this year.

Simply compare models trained on the same dataset to see how your algorithm improved.

It might be fine-tuned hyper parameters, or a whole new algorithm. It doesn’t really matter.

Let your team make the decision.

Feature Engineering Team: We already said that data features can be used in many different processes.

However, I think it’s best to measure new features as part of the Machine Learning model. This is the easiest way to isolate their real impact on performance.

You do that in a very similar way to the above.

But instead of comparing models trained on the same dataset, you compare the exact same model (say, your current one) on two different datasets.

The first dataset will contain the previous features version, and the second will contain your current one.

Seen improvement in the AUC? You can attribute it directly to the features and the team that developed them.

Fraud Operations: Break down your performance KPIs to flows/segments.

Individual teams/members should each have ownership over at least one of those.

As usual, you can segment by geography, products, payment flows, user categories, merchants, etc.

Side note: Wondering which KPIs you need to track? ​I got you covered here.

The main idea is to build familiarity and expertise that allows your team to customize their decisions to that specific segment.

In B2B, it will also serve to have dedicated fraud leads for key accounts.

Of course, you also need to establish processes (and culture!) that would enable the FraudOps team to make an effective use of the deliverables of the other two teams.

After all, if you have a new highly-performing model, but the team hasn’t changed how they use its score in their rules - you’re not going to rip its benefits.

Side note: The way to represent this in your KPI tree would be to introduce the concept of Model Participation: How many of your decisions were directly impacted by AI. But this is a whole new topic which I’ll save for a future newsletter.

Goals Over Projects

If you want to increase accountability in your organization, start with setting team-level and employee-level goals.

Do not cheat and set goals such as “Deliver Project X”.

Instead:

• Articulate what is the effect you’re expecting to see if Project X will be delivered.

• Let your employees suggest what’s the best way to achieve said effect.

• Approve projects by estimating their ROI.

Fraud is a dynamic problem to manage.

You want to make sure your team is making informed, timely decisions while you’re free to plan next year.

Have questions or feedback? Reply to this email, I read all messages.

In the meantime, that’s all for this week.

See you next Saturday.

P.S. If you feel like you're running out of time and need some expert advice with getting your fraud strategy on track, here's how I can help you:

Fraud Strategy "Power Call" - Book a consultation call with me to get clear, actionable recommendations that fit your budget. Guaranteed.
​Book a Call Now >>​

Fraud Strategy Workshop - are you an early-stage Fintech that needs to move fast and with confidence? Book this 1.5-hours workshop to get instant insight into your vulnerabilities, optimization opportunities, and get clear actionable recommendations that won't burn through your budget. ​
Book Your Workshop Now >>​

Fraud Strategy Transformation Program - are you a growth-stage Fintech in need for performance optimization or expansion of your products offering? Sign up to this 6-8 weeks program, culminating in a tailored made, high-ROI roadmap that will unlock world-class performance. ​
Schedule a Call Now >>

Enjoyed this and want to read more? Sign up to my newsletter to get fresh, practical insights weekly!