Introduction

Hi everyone, I'm Matt, I'm the VP of Engineering at Finamize and as Josh said I'll be talking about scaling content production using LLMs and how we've done that in-house.

So yeah we're Finamize, I'm going to be doing a demo but before I did that I wanted to give a bit of context around the problem we're trying to solve and just briefly introduce who we are and what we do.

So our mission at Finamise is to empower people to become their own financial advisors. And we do that by focusing on investing. So we've built a suite of digital tools to allow people to become smarter investors or even get into investing in the first place.

One of the kind of core parts of our product is our content. So we produce a lot of audio and text content in-house, which populates our products. And it's that written text piece that we're going to be focusing on today.

We've also recently launched Finamize for Business. So this is helping businesses, particularly in the finance space, engage, educate and empower their users as well.

Content Production at Finamize

So again, to help frame this problem, I just wanted to kind of introduce our content production process so you can see why we're doing this. So we have a fairly unique process and it's all driven by the fact that we've got a very high editorial bar for our output. So even if we're explaining like a simple or a complicated financial concept, we want it to be easy to understand, entertaining, fun to read, et cetera.

The Content Team Structure

So the way we do this is we basically separate our content team into two groups. So we have our analysts. These are our finance experts. They're former investment professionals.

Their role is to produce these smart, unique insights. And then we have the editors who are our linguistic experts. So they're almost like trying to act on behalf of the user to make sure the content's like fun, well explained, there's no jargon, et cetera, so that anyone can approach it.

Optimizing Content Production with AI

So this works very well for us, but there is like a lot of back and forth and time spent drafting in the middle. So the question for us over the last few months has been, can we optimize parts of this using all the new AI tools available?

So this was our idea.

So I think we're very clear on what these two teams are bringing in terms of like unique human traits that LLMs can't replicate and what they're bringing to that process. But there's a lot of stuff in the middle that LLMs are quite good at. So like drafting, writing up different versions of the article, et cetera.

1So the idea was what if our analyst puts forward the substance of the article in like rough notes and then we get AI to draft something approximating the final article and so the editor has to spend less time doing what they do and really they're just kind of looking for mistakes and adding a bit of flair to the language, adding human personality, wit, etc.

So we gave it a go and it was the usual familiar problems. So we had lots of hallucinations. It was a long way below our editorial standards, so it didn't actually save us time. We had to rewrite it.

And often it would just ignore direct instructions and things that had been told.

The Multi-Agent Flow Concept

So what we ended up moving to was this concept of a multi-agent flow. So, I mean, this might seem complicated if you're not familiar, but you can think of each agent here as just like a chat GPT window. So what's going to happen is each agent is going to have a conversation. We're going to take the output and we're going to pass it over to the next agent in the flow.

And the advantage of this is that then each agent has a much like narrower range of responsibilities They're less likely to make mistakes And just generally like quality each point should improve it also ends up being cheaper because you have like less context basically that you're passing through the whole flow and you start again at each stage and

So in terms of our process, we take our input notes, we pass over to something called the note prepper. So what that's gonna do is outline the key points and that becomes the kind of outline for our article. We then have the writer take those key points and drafts the first go at the article.

The editor just focuses purely on language, so it doesn't restructure anything. And then we have this QA agent. So we'll come to that, but it's checking for a few things. And really what it's doing is checking the output versus what we put in and giving it a score. And then we can either reject that and start again, or we can even flag up feedback for this editor and where to focus their attention.

Demonstration of the New Process

So I'm going to jump across at this point to a demo. See this? There we go.

So yeah, I mean, ostensibly, this is a technical demo because it's in code, but most of the learnings I'm going to go through are around prompting and technique. So I'll also show you a no-code way of doing it at the end. So hopefully, even if you are not familiar with the syntax and stuff, this should still be useful.

So I'm going to kick this off and we pray ChatGPT is working. There we go, just while I talk.

Yes, good point. Is that better? Yep. Cool.

Content Creation Pipeline

So this is our content creation pipeline. Our starting point is those input notes. We also pass in today's date.

And the reason we do that is because by default ChatGPT is quite bad at getting the tenses right. So it often like says stuff's happening in the future when it happened in the past, et cetera. So you'll see that we remind each step of the timeline.

And each agent has what's called a recipe. So you can think of a recipe just as a series of like predetermined prompts that's going to define the automation.

Note Preparation

So we're going to have a look at our note preparation recipe. So you are a financial analyst skilled in text editing and copywriting. Remember, your role is to provide information, not financial advice.

I'm going to give you some notes. I want you to extract the key points and then we give it the date and the data.

So at this point, it's going to have a go. So it's going to have an attempt at doing what we asked. And what we're going to do when it comes back is we're going to get it to think through how it could improve that result.

So based on the original notes, are there any additional details that would enrich the key points? And this is a version of what's called chain of thought prompting. So we're getting it to think through and improve on the answer it's given. It's very effective.

And I think one thing worth experimenting with is like the number of times you do this because it can keep getting better. We found in our case one works fine.

And then we're going to say, okay, incorporate these details into your notes. Keep it concise. And we want it to output an array of strings. So one thing we do throughout this is kind of return a structured format because we found it's more reliable. And it also means that if we want to do programmatic things with it, we can. So that's going to then get sent into our writing recipe.

Writing the Article

So there's a bit more going on here. So you're a writer. Your task is to transform detailed notes into a comprehensive, engaging, finamized style financial article.

Follow these rules. So we've got the date and then we're going to give it the general kind of like writing guidelines we want it to follow. So clear, concise language, American English, things to avoid.

We're also going to give it some examples of what we want. So there's a certain structure we're looking for in terms of these articles. So we want a title, we want a subtitle, and then all of our articles are broken up into these headings to make them more digestible.

So what's going on here? What does this mean? Why should I care?

We're going to give it three examples. And again, I think it's worth experimenting with like how many of these you give, because often it peaks at a certain point in terms of quality. And then we're just going to say, okay, have a go and output Jason. That then is going to hand over to our editor.

Editing and Quality Assurance

which we'll take a quick look at. So as an editor, your primary responsibility is to transform an article draft into a well-written article adhering to the guidelines.

And then you'll see that this, the rest of this prompt below is very similar. So we've got the same language guidelines. We've also given it some examples, different ones, but still very similar structure. And then there's also a very similar prompt.

So I think what's interesting here is 95% of this recipe is exactly the same, but what we've done is we've split out the responsibilities into two steps. And so it's more likely to be competent and like give a good result and make less mistakes. And we found the overall process works much better. So that's an interesting thing.

so then we move over to our QA that I mentioned so we do a few things here One thing to note is that we do quite a bit of this programmatically, and that's because we found we get more reliable results than purely through AI in some cases. So for instance, we check the JSON structure, we add some formatting, so our CMS uses Markdown. We apply that to the titles using code because we found out ChatGPT makes some mistakes.

4.0 might be better. We've not tried 4.0 with this. And we also do a fact check. So this does use AI. This is taking the input and the output and making sure that we haven't misrepresented any facts or whatever. But I think you can apply, you can add any checks here that are critical for your use case or your business or any edge cases that you want it to pick up.

So while I've been talking there, we should hopefully have an output article. So you can see here, this is a pretty good draft. So we've got a title, we've got a subtitle, we've got all of our headings that we're after. It's pretty well explained. It's got a good structure. And this is, for an editor, this is a lot less work. So all they're really doing now is like sprucing up the language and making it a bit more entertaining and not getting into like drafting and redrafting and back and forth, et cetera.

um and i should say uh for us like this is one kind of content type we've also got kind of more like commercially valuable types of content that we write they require a lot more input from our team um and so what this allows us to do is have our staff work on things um that are much more demanding of their time of their skill and spend less time doing things that they don't need to be doing so that's how we see this is like how to make the best use of people's time

No-Code Solutions

So I did mention you could do something very similar using no code. So we've done something very similar as well in Zapier. You can also use Make.

And what happens here is we just have a Slack channel. You can post an input message and it will output the result at the bottom. But we've literally used exactly the same prompts that I just went through.

So this is our note prepper. And you can see almost word for word we've recreated it here. So I think the point is, this doesn't have to be like a technical endeavor. I think it's about knowing the playbook, knowing the techniques, and just experimenting with what's best, breaking things up into smaller steps.

And you can build some pretty sophisticated pipelines and processes without touching any code. So I will end on this.

Conclusion

So yeah, I mean, if anyone's kind of working on things in a similar space, please do get in touch. We'd love to share learnings in more detail and hear what people have been up to. I think we're in a new space here. I think the only way we kind of move forward is if we all share learnings about what works.

And yeah, similarly, if you think this content could work for your business, please do also reach out. But yeah, that's it. Feel free to ask any questions.