Introduction

Okay, thanks Melinda and the whole Mindstone team for the invite.

Overview of Bank Reconciliation with GenAI

Today we are going to talk about bank reconciliation with GenAI. So my name is Pablo Manzano. I'm the lead data engineer at MBAT.

Understanding GenAI

So yes, well, we have two concepts here, bank reconciliation and gen AI. I guess you all know what gen AI is, right? That's why you're here.

But yes, basically it's the part of AI which can create new content. Right now, I think we are a bit outdated because we do gen AI, but just for creating structured data.

Right now, the hype is in multimodal, Gen AI, right? I guess you've heard about it already, where you can mix video creation with text input, whatever.

Explaining Bank Reconciliation

But yes, I'm guessing not all of you have heard about bank reconciliation. So what's the bank reconciliation? We can simplify this.

About MBAT

So let me quickly introduce us, EMBAT, or what we do. We are a Spanish fintech. We've built a software as a service for financial teams of medium and large companies.

And we automate payments processes, treasury and debt management process, and accounting and bank reconciliation, right? We can say it's an exercise all financial teams need to do in their companies in order to prepare their financial results and to reconcile the payments and collections they do from or to their providers or clients, right?

MBAT's Real-Time Data Integration

At MBAT, we connect in real time to our customers' banks and their ERP systems. So from the one side, we get the bank transactions, once again, in real time. And from the ERP, we get a lot of different data, such as, well, mainly the invoices, their counterparties, and other variables they use for tagging all these documents, right?

And this Information coming from the ERP, it's super rich because they can customize all these variables, all these tags. They can assign to all their invoices who their counterparties are. But in the other hand, from the banks, the information we get is really poor.

The Challenge of Bank Reconciliation

Mainly we just receive the date, description, and amount of the bank transactions, right? So part of this accounting or bank reconciliation problem is to match their payments or collections in the bank with the invoices they have registered in the ERP. This might sound quite simple, right? So if you have bank transactions of whatever, 1,000 euros, and then you have the same invoice in your ERP, well, you just need to match them, right?

There's no AI behind that. It's just simple logic. However, in real life, the possible relations are way more complex, right?

I can tell you about a few examples, which makes the relations are not just one-to-one, but there can be one-to-zero. This means one bank transaction having no invoice registered in the ERP. For example, if you're paying the salaries of your employees, you don't have an invoice for that payment. And then we can have end-to-end relationships, meaning, for example, one provider paying all the invoices they owe you in one single payment or the other way around.

And you can also have, like, you can pay an invoice in different stages, right? First you pay 50% of it, and then the other 50% at the end of whichever process. So yes, these relations can get super complex. And in large companies, you can imagine they have thousands of invoices on a weekly basis.

So the data sets you have are kind of large. They can scale a lot, depending on the company size. And the relationships are not so simple, as I already mentioned.

Solving the Matching Problem with AI

So you can already imagine that this mapping problem, it's not a classic ML problem, right? We've investigated a lot a lot of documentation on classic ML, but there is no ML algorithm that can solve this matching problem. So the way we do this is by tackling different use cases, right?

So as I mentioned before, there's a quite simple case where you have this transaction and the invoice with the same amount, same date, that's super simple, no AI behind that.

But for these kind of difficult relations, it's where we use the GenAI. And let's see how we do this.

Enriching Transaction Data

So as I mentioned, the information coming from the bank, it's super poor. So in order to facilitate this matching, we need to kind of enrich it or structure the data, right? Remember, we just have the amount, the date, and the description.

The description, as you can imagine, is a string, like a text. So you can imagine that, OK, it's unstructured, but maybe processing that string, you can get more kind of structured data, right? You can have who the counterpart is, because, for example, the description is factura to this proveador or whatever. And you can have a lot of different data which you can map to the information coming from the ERP.

For example, the invoice number, right? So the same, invoice number 101, whatever. So you need to map this.

Adopting Natural Language Processing

So you can start thinking about, natural language processing models, which you have a bunch of pre-trained models which solve already this problem. To be fair, we did a proof of concept with them, but didn't work so well, at least without a lot of fine tuning.

because the transactions coming from the bank, the structure of the description can vary depending on one bank or another. Some banks will send you the invoice number, but other banks won't, different keywords or whatever.

Choosing Large Language Models (LLMs)

The reason why we tackle it with an LLM was that by the moment we developed this, by the way, this was like, not even a year ago, I think in Q3 from last year. We were like 30 employees at MBAP, and the data team was just me. So as you can imagine, I had a lot of different tasks to do. I couldn't focus 100% of my time on this problem.

So the LLM was kind of plug and play. We will go into some details there, but kind of plug and play. We developed this, well, it took a while to develop, to be honest, just because the solution we deployed didn't actually exist when we started tackling the problem. We used Google's Palm model in batches and with another tool we'll talk about in a bit. And when we started deploying it, developing it, or part of these tools didn't exist. So the process was a bit longer, but if I had to deploy today, it would be hours of work, literally plug and play.

Why Google's Palm Model was Selected

So yes, that's why we tackle it with an LLM. I already mentioned that we use Google Spam, and there are multiple reasons for that. The first of it is that we already use Google Cloud tools. Our solution is cloud-based, and our infrastructure is in Google Cloud.

So it's kind of easy to connect your data source with this LLM inside Vertex AI. But then another super important reason, actually, the first one, it's about cybersecurity, right? So we have the terms and conditions of Google's models, and we are 100% sure that our data isn't used for training other models or whatever other reasons. Our data doesn't go out of our organization, right?

And as you can imagine, we are a fintech, like financial sector is super, well, they have a lot of regulations, so we cannot use other solutions that can use our data to retrain models or whatever, right? So those are the main reasons.

Actually, right now that we're in Google campus, we work here for a year, more or less. And they have a super cool program for startups. So we also had some credits to the proof of concept. So that's also another reason, right? If it's cheaper, obviously, you would go with that.

Implementing LLM for Data Structuring

So, well, yes, as I mentioned, this is plug and play more or less, but it's not just, okay, here you have my two data sets, transactions and invoices, do whatever you think. As I mentioned, it's about like getting structured data from unstructured data, right?

Here I plot some results of our problem. The LLM model responded, basically what we did was, okay, for this set of transactions, can you structure this description and give me who is the counterparty, what's the invoice number, and many other variables we use for the matching, as I mentioned. The model did respond for 91% of the transactions, but however, this response sometimes is really vague, right?

It could say, okay, I couldn't find what the counterparty is, and that's like, OK, thanks for your job, but I'm still at the same point. So depending on our clients, we could, on their sector mainly, and how they do their payments and collections and so on, we could then reach between 40% and 70% of their data set.

And this drove to a 15% improvement in this automatic reconciliation rate. This 15% also depends a lot on their business and how they do their payments, because for some of them, which they have, for example, 90% of simple one-to-one transaction and invoices.

There's not a lot more to do, but there are some really cool examples where we've plotted with them how we've improved their automatic reconciliation, and it's amazing, actually. So, yes. As I mentioned, this is not just plug and play. You need to work on both the input data and the output data.

Prompt Engineering and LanChain

Here, I want to talk about prompt engineer. For sure you've heard about that there are new roles or new companies offering prompt engineering roles. I have mixed feelings for it because for the one side, obviously, it's super important, but it's not a huge deal, right?

Actually, this morning I've been in an event at the Google offices where they talk about Gemini. I'm not sure if you've heard about it. I've seen some faces doing like they've heard about it.

They handed out this prompting guide 101. It's Gemini for workspace, they've been showing really cool examples on how they've ingested Gemini in their workspace. So Gemini actually writes an email for you a document, whatever, really cool stuff.

The Role of Prompting in AI Success

And basically the main part of the conversation was, okay, if you want to have a successful experience with Gemini, it's all about prompting, right? You need to give it some context, ask for the task, the format and so on. And here is where LanChain comes in.

LanChain's Role in Application Development

I mentioned here in the first point what LanChain is or the definition you can find in their web. It's a framework for developing applications powered by language model.

They have a lot of tools. Actually, I haven't checked it recently, but I know there are a lot of different tools right now in the library, but we use it for basically data input and data output format, right?

You can pass via line chain, whatever, BigQuery table or a CSV file. And with this library, you can format the data so the LLM knows what your data is about. So yes, this is the main tool we use for it.

Scaling the Solution

Oh yes, about the scaling. We process, right now, about 1.5 million transactions a month. So, as you can imagine, this LLM, well, it's basically like asking ChatGPT, okay, what are my descriptions, my counterparties for these one million transactions? Obviously, it won't, like, it will get crazy and won't respond to anything.

So we also work on some pre-categorization of the transactions. Remember I mentioned this example where you want have an invoice for your salaries, payments. So actually you don't need to pass that transactions, right? And you can reduce the data set we are passing to your LLM, like in 60% of the volume.

Recap and Reflections

So as a recap, Bank reconciliation, it's not an AI problem, but AI will solve, will improve your deployment on the automatic, and it's all about how you build your chain of data, how you process your data, and how you process your output, right?

But yes, as I mentioned, always when I talk about this, the questions are like, okay, and how much does it cost to do all this process? It will be plug and play as of today. But yes, and I'm guessing right now there are a lot of improvements on it, which we haven't tackled yet.

But as of today, it works really good. And I think that's all. I'm on time, more or less.

Conclusion and Q&A

So if you have any questions now. Thank you.