AI Augmentation for Knowledge Businesses

I spoke about how to build stuff with AI. I'm not going to talk about that today.

I want to try, I wanted to challenge myself and see, could I talk about something really nerdy without getting too technical? That was my challenge to myself, and you can tell me afterwards when we had pizza, if I failed or if I managed. managed.

I'm going to talk about, and I think Avik talked about cost versus growth. I like to talk about augmentation a lot. I'm going to talk about augmentation a lot here as well, where it is not about how do you save as much money as possible using AI, but how can you augment the resources that you have using AI.

Now, having said that, I'm going to talk a lot about consultancies today, or what I call knowledge businesses, so anyone who works with, sells what they have in their brains.

I think it's fair to say that the 814 ,000 people who work at Accenture, there's not going to be that many people working at Accenture in the future. I think I can go out on a limb and say that's not going to happen.

And these companies, and I'm not talking about Accenture specifically, these consultancies, agencies, they're struggling today because they're trying to figure out how can we use AI to augment what we do without losing the revenue and that revenue is usually based on what? Hourly rates.

Hourly rates is the worst way of taking, getting paid of course, but it's even worse if you're using AI because your customers are going to come, your clients are going to to come to you and say, hey, if you're using AI and you're saving money, I want to save that money too.

Okay. Sorry. I'm kind of jumping into what I'm talking about. So that's what I'm going to talk about. Okay.

From Creative Agencies to AI Workshops

My name is Eric. My background is way back creative agencies. I was a creative director for many, many years.

And in 2017, I said, nah, I don't want to do that anymore. It doesn't make any sense to me. Why am I just trying to talk about stuff instead of fixing real problems?

So I changed to, and I've worked in FinTech and I've worked in other startups. So now I do some consulting. I do a lot of workshops.

I did a workshop this weekend, weekend, week ago, for a group of risk managers. These risk managers had never built anything in their entire life with computers. Never. They never made any software product, ever. ever.

In one weekend, I showed them how to build a platform. I showed them how to build with AI, and all of them, all 15 of them, had a product at the end of that day. A very basic prototype, of course, but it shows what we can do with AI today.

That's not what I'm going to talk about today, so I'm going to just move forward. it.

The Knowledge Challenge in Consulting

So knowledge businesses, they sell what's in people's hands. That's a great business because AI can't really replace that. If I know a lot about your business, AI can't easily replace it. It can augment it, it can help it, but it can't easily replace it.

But there's so many things that's based in that information that they have. Their briefs, their client history, history, all of that judgment that they have, that they like to talk about, and it sits in the threads, in the decks, and just like people's heads mostly.

And this is not just consultancies, it's creative agencies, it's VCs, it could be architects, lawyers, all of those companies have the same problem.

right so they move they have generic knowledge which is like we work in that industry so we know a lot about that they have a lot of personal knowledge I know this because I sat in a meeting and I talked to that person right and you build up that you know the knowledge base that sits in people's heads there's knowledge about their client and then this historical knowledge and that is is really hard to get by because all those documents that have been produced over the years has never been used to anything other than summaries of meetings and whatnot. So it's a huge, huge challenge for these companies.

They're trying to offer value without becoming generic. And that knowledge leaves the company as well. First and foremost, people quit. it. People change jobs. So when someone walks out the door, that knowledge, that relationship, all of that tends to just walk out with it. It's very hard to kind of have a handover of any thing of value, right?

There's also too much information. Sometimes you just have too much and then people just forget. So that knowledge isn't stable, right? And AI can't really replace it the way that we use it today.

Just-in-Time AI Augmentation

So I'm going to talk about what I call just -in -time augmentation, and it's about routing each task to the right model and retrieving the right context for the right person.

I'm going to ask a question before I do that. When you use AI, who here has a clear idea which model to choose anyone anyone feel that I have a good idea I'm gonna use a this model for this task you do good most of us do not and most of us should not I see I

think when you have it let's think a corporate where they have 30 ,000 employees whatever probably five of those people are capable of sitting down and go okay I'm gonna use sonnet 5 .5 on this one I'm gonna use Opus on you know it does not make sense.

So I think in the future we'll see more routing of that and that's really what we're gonna get to.

A System for Capture, Retrieval and Delivery

Okay so what I'm talking about here is a system that captures information, it learns from that information, it augments the output of that information and it delivers that information, right?

And we have this way of choosing models so we can either choose one model for everything, we just use that model, or we use Claude or ChatGPT and those sub -models that sit within there, and we use all of that for tagging a strategy, and cost then grows with every query, and we just change it when a new one comes in, and we just go, okay, there's an Opus 5 .6 now, so let's do that kind of thing, right?

The other way of doing it is what I call just -in -time augmentation, which is matching each capability to the specific task, but you have to combine it with speed, cost, and depth, set per step. So, cost follows the difficulty of the work, and models swap by configuration. So, this becomes automated. I'll try to explain this.

So, the first thing we need to do is to triage. right so we need to figure out what is the task that we're solving okay so if the AI is doing this we cannot rely on a little traditional LLMs to do it so we have to have other types of model that will very easily find out what is a task here what am I supposed to do with this information then go okay with this information and this happens in milliseconds so this is complicated in in this part, I need to decide where this goes.

So then you have to retrieve all the information that you need to make that reasoning and that decision that you make. And then that goes into the reasoning of the model.

And then on top of that, you should actually have a model that verifies the output as well. Because we know that AIs hallucinate, and they make up stuff, and we don't know if the output is 100 % correct, right? So this is what we're building towards here.

Routing Tasks Between Fast and Reasoning Models

So we are then talking about two different systems. One is a system one.

Anyone seen the presentation of Jev, the new Jev model? You have? Anyone else? It's pretty cool. So Jev is a system one type of model.

model, it does not give you any answers. But what it does, it figures out.

So you can take the video I saw, and I've seen many of them, but the video that really caught my eye was they took 56 ,000 or something emails, and they wanted to find out how many of these emails had a negative starting point. So they just went through, and this in seconds, and cost less than two dollars I think to turn through all of this.

With traditional LLMs this would take minutes, maybe tens of minutes, to kind of process through and figure out and it will cost a fortune.

So these models are really really quick and they just give a percentage of likelihood rather than an actual output.

And then you have system two which is what we traditionally think of as LLMs, large language models, they reason, they think, they give an output okay five minutes okay I have to hurry up now okay so I'm

gonna be very quick on this one because I've talked about this now small and fast is does the triage and the check deep reasoning comes with the bigger models and then you have long context and the context has to happen let's see if I can get to that slide I'm gonna just skip a little bit okay so do you

Building Context-Aware Retrieval

know what a RAG is? Retrieval Augmented Generation. So RAG is basically an AI that has the context of your company or of your data, right?

But they are very generic as well because they read all the data and they have no way of sorting it. So they can answer specifically based on your data, but for a consultancy or for an agency, it would be very hard because there's no walls between the clients, the chunks of AI chunks and the context windows, they run out and you don't have that information necessarily.

There's no memory of outcomes in the rag and similar doesn't mean that it's relevant. relevant.

So to do this, we need to then have any source in. So it's data agnostic ingestion. You have to take all this data. You have to structure this. And this is really the key here. Structuring unstructured data becomes super, super, super important.

And to understand that is like you have thousands and thousands of documents. But if you're going to search through those documents, that's going to take time. So you need to speed that up. right and that makes it ready to retrieve and then you have a semantic layer that gives that retrieval a map and I'm not going to talk to that because I want to talk about this again on top of this we then have to add six

The Context That Shapes Better Answers

in my my opinion six dimensions of context because the context makes the the difference of what the output is. So what is this firm? Who are we as a firm? What's our culture? How do we work? What do we do? What's our industry? Are we an architect firm? Are we a consultancy? What are we doing?

Who's our client? What are their quirks? What have they said yes and no to before? What's the project that we're working on? What's the budget? Who's working on this? All of that information.

And then most importantly, perhaps, is who's using this? Is it a managing director? Is it a creative person? Is it a project manager? They have all different needs, right? And then when is this question being asked? So when are we querying?

Am I already running out of time? No, you're good. OK.

So for example, you can find out what did we learn from last pitch? The creative director wants to know what the client loved about the concept or the tone or all of that.

that, but the managing director, or the account manager, sorry, he wants to, or she wants to know who approved it, what was decided, all of those technical things.

Open-Weight Models, Cost and Privacy

And to do this, my proposition is to do this not with API -based LLMs, but actually using open -weight models. And the reason why I'm using open -weight models, anyone want to guess?

Oh, are you saying something? Hmm? It's free. Well, it's not free, because you have to pay for compute rather than tokens, but yes, it's much cheaper. So it's free if you run it on a $100 ,000 computer, it's free. Yes, any other reasons?

So one reason in Europe, for instance, privacy. Privacy. So in Europe, they have this new EU act, the AI act. And it's very strict in terms of what AIs can do, et cetera. They don't like Chinese models. They don't really like American models. But Europe only has Mistral and a couple of more, so they don't really have models. So this is a problem. But open weight models allows us to do.

I think the models become less important because they'll be so good. they'll be so good that the question isn't is that model good enough the question is how do we root our queries in a way that gives us a right response in the right time at the right cost and cost becomes sorry but cost here becomes really important because if you're paying you know thousands and thousands of dollars every time every day for for API costs people are going to start looking and say hey we need to figure out a better way of doing this we can do that with with a better retrieval system and a better set of of models I think

Guardrails and Pseudonymization

I'm done I think I'm just talking on I wanted to talk one more thing is guard race becomes also crucial in these things I like to talk about

pseudonymization so for instance if you ingest a conversational slack let's say you have all your all your 800 ,000 employees on slack and someone says you know that Eric guy that's me he's a proper douche right I don't want to be able to go and say to other you shouldn't have said that was not cool man right so I can't we have to separate who said it from what's being said so So you need to have a system that takes privacy really, really into account in these systems.

And I think most companies that are looking into figuring out how to do AI, they're not thinking along these lines at all. They're just going like, okay, let's just get some subscriptions to ChatGPT or whatever it is. or as the corporates, you guys in corporate, you like co -pilot, right? Co -pilot is the corporate answer to AI models.

I'm sorry, I lost my slide, so the last part was a bit... Everyone listen to your words.

The Case for Vertical AI

Yeah, so what I'm trying to talk about is to build something that is a retrieval system for these, or a vertical AI, if you will, for these knowledge businesses to be able to, in every single moment, get as right answer as possible.