Good evening everyone. Thank you for making it today.

Token Economics and AI Costs

I would like to talk about the caveman skill this evening. Some of you may be familiar with it, but I'm going to do so in context of token economics.

Tokenomics unfortunately as a phrase has been stolen a little bit by the cryptocurrency element, but I'm going to steal it back because I think it applies to AI even more than it does to crypto. And the main reason for that is because it's all about tokens. tokens, tokens, tokens.

That's how the LLMs that we work with and interact with right now today work. But that's also how they're charging us.

So I'd like to talk about some ideas as to how we can manage the tokens in a more efficient manner. And token efficiency is of growing importance for individuals who don't want to necessarily move to the next tier and the subscription layer and of course to organizations who want to keep tabs on how much they're spending on AI and make sure that they're not burning too many budgets in trying to keep it up and running.

If you come in, join us, welcome.

What Tokens Mean for AI

So a quick word on tokens, what does it mean? Well I did what most people do, looked it up in the dictionary and you've seen all the different definitions I'm sure before, and I don't think many of these, if any, apply really to the concept tokens in AI.

So of course the dictionary needs updating, but one that really struck me was the idea of an IOU. I -O -U. Of course, when it comes to AI, arguably it isn't any more about IOU is about you owing AI.

And the reason for that is that we've seen all the horror stories about how expensive AI can become. Now this perhaps is a horrendous example and I'm still not fully convinced that it's a real one because the numbers involved are just ginormous.

But we've all heard of experiences of organisations and individuals having what I would best call bill shock when it comes to AI. What I'd like to look at is what we can do to actually control that and manage that a little bit better.

And really this is about token efficiency. Very simply how we can use token consumption in a manner that still delivers us the same results but without burning the bank.

And it possibly starts getting very technical very quickly but I'm trying to keep it high level because because obviously we've got mixed audience here. For those of you who want to get down into the weeds, more than happy to take that offline.

If you want to go further still, perhaps I'm not going to be the right person to discuss it, but I can do it to a certain layer.

How Context Windows Drive Token Use

On the left -hand side is what we normally start out with. Each one of us goes through this learning iteration. When we first get exposed to AI, we start off with a simple window, usually with ChatGPT or perhaps Claude Desktop. stop, we ask it a question, we get a response back.

Very quickly though our usage matures and we start doing more and more with it and also the tools mature as well and they start doing more and more in the background with or without our knowledge. So typically we quickly get into this environment Where we are providing, yes, a question, a message if you like, but also, as well as the system prompt that comes from the vendor, we have all kinds of other contextual information being added in. These could be documents.

Sorry, has my microphone fallen off? My apologies. Is that working? That's if I turn it on. Can you all hear me okay? Right, my apologies. is.

Hidden Context and Conversation History

Right, so very quickly we get to the point where we actually need to understand everything that's being added into that context window that's passed through the LLM. Why? Because this consumes tokens, input tokens.

So every tool that we connect in, so if you're using an MCP server, if you connect into the file system, this is all supplying a long list of actions that it can do that adds to the context if you're attaching files those also get added in so that it can work on them

it is a stateless protocol so just like http if you're familiar with that web protocol from one page to the next it forgets everything it has to be told where you are in that context that's why we have cookies in web browsers in order to be able to help retain some of that that session history.

When it comes to working with an LLM, the way that it happens, it contains or remains contextually aware, is by providing that context to it every time. Then it formulates the response, that gets added on to that message history, and the cycle repeats.

So every time that you're asking it a question, all of the previous conversation, all of the previous files, all the previous tool outputs are also being added in at that next turn. The more turns you do, the bigger the queue, the more tokens you can see in.

Context Limits and Compaction

So in a simple context this is what happens. You start off with a user message, you get the response, these get added together, then the next round another user message, more output, and you can see very quickly it grows until you exceed the context window.

Now depending on which model you work on, that happens to be a finite number of tokens. In some instances it could be 200 ,000 or it could be 1 million. But sooner or later, as you have a longer chat and you're doing or with it, you will hit that context window.

My question to you is what happens when you hit that context window? Compaction occurs. Yes, compacting this chat, it will come up in a variety of different messages and styles depending on which agent you use.

And what happens when you have one of those compaction events? Yes, effectively it's almost as if it's gone dumb overnight. All that intelligence that has built up through that contextual awareness has pretty much been thrown out the window and you're dealing with almost a very dumbed -down version of the AI again as you have to rebuild that awareness and that context.

That's why sometimes it's easier to start a fresh new chat and resupply all the context and start almost not exactly from scratch but but again, in order to work through a complex problem. So anything that we can do to reduce the amount of tokens consumed helps us in dealing with the AI.

And by the way, I'm saying AI, AI, AI, by that I mean LLMs in particular, because that's what most of us are working with. There is a little bit more going on behind the scenes. Yes, it is the user message and it is the response, but that tool use, the list of tools that are available, the system prompt, and also its reasoning are typically involved as well.

If you're actually programming at API level, you can actually choose whether or not to include the thinking in some models. Nowadays, you tend to find that that is included automatically. So as you can see, the chances of actually hitting that limit are a lot higher than just based on the input and output. A lot is going on behind the scenes.

Strategies for Token Efficiency

So the four main ways that you can try to reduce the token and burn you can actually look at the prompt how you ask it the questions your user messages and there are a number of different techniques i won't go into here but some of the most common ones are creating like macros for example so you don't repeat yourself so much and you reduce the size of the input you can also help reduce the number of tokens consumed by thinking a little bit more carefully about how you use the context and supply the context if you're using a rag system, for example, that's a great way of actually reducing some of the different token consumption that takes place.

You can also work on the differences from one to the other, the deltas, and that's another technique, and so on.

Why Output Reduction Matters

If you're looking at the architecture, there are a number of different techniques as well that you can use there, but what I'm going to be talking about here next is the output. And that really is about trying to limit the amount of output that it's producing at each turn.

Now why the output? Good question.

We're limiting the output and in this case through the caveman skill because output typically costs three to five times for more modern frontier models that's about five times the cost of the input.

So if you're going to make any savings anywhere it makes sense to do that very much on the output. If you can reduce the verbosity The amount of words it uses, it reduces the amount of tokens consumed.

And of course, over longer conversations, these kind of savings compound. And so if you're doing it properly, you will end up with anywhere between 50 % and 75 % savings in terms of the token utilization and therefore can have longer, more meaningful conversations and reduce the token usage and therefore cost in the process.

The Caveman Skill

And this is what the caveman skill does, as written by the caveman skill itself. You will notice it removes all the extraneous terms, the ands, the ors, and it just tries to get to the key context, all the key matter. Anything that is It's superfluous, wordy, yet strict. No waffle, just the key facts.

It sounds really basic and stilted, and it sounds a little bit like a caveman would speak, hence why it's called the caveman skill. Very straightforward to use.

Has anybody actually used it in this room? Can I have a show of hands? Couple of you, great. Do you use it all the time, or do you use it occasionally? All of the time in your case, and for the rest of you, occasionally. Okay I think that's the best way to go about it.

If you just bear with me one second I'll just switch across, excuse me I'll just put this down.

A Practical Demonstration

And has anybody got a good question to ask Claude this evening? No? Okay let me improvise then.

Explain to me the story behind, oh let's relevant film. The Terminator. That seems quite relevant for an AI audience, doesn't it?

So by default you can see that the answer is comprehensive. Some would say a little bit waffly. It gives you the plot, it explains what takes place. I think you're all familiar with what that story is, so I won't spend too long explaining to you what The Terminator is.

As you can see, there are quite a few paragraphs, lots and lots of words, lots and lots of tokens. And now I'm going to ask it to do the same thing but using the Caveman skill. I have to admit this is going to take a few seconds longer than normal. If you were installing the Caveman skill in Claude or as a GPT in ChatGPT and so on, this would be a lot faster. The reason it's going to take a few more moments in my case is because I've got it hooked up to a Notion database system, which is another story itself. But as you can see, here we go.

Already you can see that it's pulled a lot of the words out of that definition, and it's come back in a very terse manner, but it's explaining what's going on. As you can see, it's readable. It's not necessarily something that you'd always want to read, and it's not something that you'd want to use for example for marketing copy or for an email to your boss perhaps however the great thing about

the caveman skill is that you can tell it what to use it for and when not to use it so you can adjust it accordingly not to use it for code not to use it for emails to your boss not to use it for marketing copy but certainly for everything else go ahead that's pretty much how it works

If I find my cursor again. Cursors. It's there somewhere. There we go.

Using the Skill Selectively

This is where you can download it if you want to have a play with it.

It is a skill that's been developed by, as you can see there, Julius Busse. Not me.

I find it useful from time to time. I don't use it all of the time. Some people do.

But try it. Give it a go. The mileage may vary, but I have found it a very useful way of reducing the output token count and therefore some of the cops.

Complementary Tools for Input and MCP Efficiency

It is one approach it's not the only approach the other one though that works alongside it really well and I would suggest that you have a look at it is the headroom skill if you're old enough you'll recognize Max Headroom there but I guess I'm the only one in this audience that's old enough to remember him.

Same idea but this one actually concentrates on the import so it tries to identify within the context what can be done to strip the number of contextual tokens that goes into the import. So you can use this and the caveman skill at the same time to get the double efficiency savings from that.

There is another one that I'd also recommend if you aren't familiar with this. Some of you may be familiar with MindStone having a tool called Rebel. One of the components within that is the SuperMCP Router component. What this actually is is an MCP server which routes to other MCP servers.

But rather than just routing, it actually collects the information from all of the different tools that you connect with and just exposes one call to the AI system. So instead of 80 available calls for one MCP, 120 for the next MCP and another 100 for the next MCP, which all consume tokens, it shows one skill or one tool available which is list tools.

So as the AI determines that it needs to call on something it says to the Super MCP hey I need something to look at Outlook and it can search through and show whether it's got something relevant for that requirement and provide it. Used in conjunction with the others it of course saves tokens but in this case on the MCP tooling.

Of course this is all

Managing Token Usage at Organisational Scale

great for the hobbyist for the consumer but at an organization level these do do help a little bit, but not in a managed and supervised way.

And I recognise that this is a growing topic for large enterprises and even smaller ones as well.

The good news is that scalable solutions do now exist.

There are a variety of different platforms and tools that can be used to track and manage token usage across organisations.

And these also start tracking the ROI from the users when it comes to how they implement AI systems within the workforce.

And I guess the relevance here is down to this last point, the actual bills and spend on AI is rapidly becoming a topic of AI governance within organisations and this is where QR security is actually focusing our efforts as well because it's part of a wider discussion on governance and security usually part of the strategy and governance discussions.

Conclusion

I hope that's been of interest. Thank you very much for your time.