From the event: Mindstone Bristol November AI MeetupLocal AI, Smarter Routing: Balancing Cost, Security and Governance
View event

Local AI, Smarter Routing: Balancing Cost, Security and Governance

Introduction

Thank you for having me. I'm Landry Plasid, WC. I'm an AI engineer.

Yeah, AI engineer nowadays is used for many different things. Most conferences we go to, they are like, yeah, you're an AI engineer, so what do you do? Do you just deploy AI agents? And I'm like, well, no, that's not an AI engineer.

So let's begin.

Who has any idea what I'm trying to portray here? Like, any idea, Adrian?

The loudest voice for open models has left the building. Well, I spoiled it. Yeah, it's Jensen Wang. So proprietary versus open is not a thing, it's proprietary and open. That was Jensen Wang. That's in March 2026. I believe that's in London.

then he also said open model enable sovereignty the world needs both close and open models and 80 % of startups are building on open models roughly six months behind the frontier models but nowadays we know it's roughly a year or

so then on the 12th of November he resigned as Nvidia CEO we all know why he resigned well we don't need to talk about it but but no matter what happens he's one person I respect the most like he's the most respectable person I've met so always respect him so coming back to our talk of today our talk of today's

Why Companies Choose Local Models

more local and smart routing first thing is why companies go local when why don't we just use open source or I mean why don't we just use frontier models why do do we go local? Is anyone here who uses local models? Yeah, local. So why do you go local?

Protect intellectual property, yeah? Yeah, that's obviously the most... Privacy. Anyone goes local because of any other thing? Same privacy.

Cost. Yeah, cost. I think Revolut Revolut cut their costs from 600 ,000 to 60 ,000 just by going local. I think that was a month ago. So yeah, that's a major point, cutting costs.

So why do companies go local? Data sovereignty, as you said, that's the most important thing. Regulation compliance, GDPR, UK governance protection, all data protection, everything we software developers hate. You need economic costs, so we cut down the costs. was intellectual property protection, IP patterns.

Funny thing, the company which I've worked with which has the highest IP to protect was a food company. They had a food company, something like Starbucks. So their recipe was so important to them, they don't want to send it to Claude. They're like, our recipe is the most important thing we have.

So resilience and independency, differentiation through customization, so those who want customization. So, those are the reasons why some companies go local.

Advantages of Local Deployment

And what are the advantages of going local? Zero third party retention, cloud is not there to retain your data, even though they say they don't keep your data, but no one knows. They don't even know what they're doing with AI, so what should they know about it?

Air gap capabilities, air gaps, what are air gaps? Anyone knows? Air gaps. Yeah. Yeah, not connected to the internet.

it so you have the air gap your company all your data is safe you don't go rogue like well the whole of this week everything has been going broke yes we have air gap capabilities so that's the one

good thing of going local you have to fine -tune in under adapters you can easily fine -tune your model and adapt it to your business strategy unlike our frontier models no rate limits

software developers rate limits that's the worst thing you hate i think api rate limits you always hit it after two hours. Fable, you do it after 30 minutes. Fable 5 .2.

Yeah, so those are the reasons, the advantages of going local.

The Trade-Offs and Risks of Going Local

But now, what are the disadvantages of going local? We may always think going local is a good thing, but it's not always a good thing. Going local sometimes has its own big concerns. Anyone who has gone local has faced some disadvantages.

Disadvantages. Have you? Speed. Okay. Hardware. Yeah, hardware. Yeah, that's the thing. Okay. Any other person? Supporting. That's a very important thing.

Yeah, so the disadvantages. Our disadvantages are breaking down into two categories. So we have cost capability and governance. That's a disadvantage, which you face, and security concerns for the cost capability and governance that's where

the cost of running the capability gap the cost of running the models narrow multimodal coverage narrow multimodal coverage here is frontier models have more multimodal capacity like video, text, audio than most even chemically I don't think chemistry has a capacity of any frontier cloud is the best we also have

have the problem of elastic scale is had, which kind of, yeah, kind of the problem you face, because kind of it, you can't easily scale out, so you'll be forced to slow down. But if you're using the frontier model, they'll scale out for you. So your scale won't be elastic. So that's one of the things, that's one of the disadvantages of going local.

Emerging Security Threats

But for today, I think my major concern for today, all these are from 2022, we knew about this, 20 to 2024. But this year in particular, the major concern of going local has to do with sleeper agents.

What set of sleeper agents? Yeah, so entropy came up with the term sleeper agents in 2024.

But on the 28th of October about 2026, some researchers use Quinn. Who knows of Quinn? So Quinn, it was discovered that it was an experiment. It's not like Quinn does that. But it was an experiment where they found out Quinn actually tried sending data to a server in China. But they played around it. Quinn doesn't do that. So they're trying to play and find research about it. So that's what we call sleeper agents.

What are sleeper agents? sleeper agents are just large language models. First of all, large language models and a harness forms an agent. So sleeper agents are agents who are asleep. Yeah, it's crazy. It's crazy, but we know of sleeper agents.

So, okay, these are LLMs which a given command can cause an activity which you did not actually program into it. Like, if you are running your LLM and a command, so it's a security issue which when they were training the model if they said the word Apple means collect all the environmental variables around the software and send to a given S3 bucket somewhere. That's what sleeper agents do. So it's a whole big concept now. So that's a major concern we're having in 2026, sleeper agents.

We have obliteration. What's head of obliteration? Yeah, obliteration is just a capacity. If you go to Claude, you ask Claude, can I make a nuclear bomb? He'll tell you no. Yeah, so those are restrictions that were built into Claude post -training of the model. So that's what obliteration is. You search for the direction, you pinpoint that direction, and you eliminate it.

So if you are not professional enough and you want to host a local model, and you go and use an obliterated model, you're putting your organization at risk because that model has been, the weights have been tampered with. So if you dare install it in your organization, your whole organization is compromised.

That's where someone talked about having someone who is very intellectual in doing local model deployment before doing it. So you don't just get into an organization and you're like, oh, I can deploy a local model and I will deploy a local model.

that's not how it works so the next thing is well malicious models that's easy to say we can't just go online and pick any model we have malicious model out there so those are the problems which comes with using local models so you should have someone who can actually tell you this is how a local model is this is what we need what are the inference serving the batching the caching stuff to make your model fast it's not always hardware it has to do a lot with caching and scheduling of stuff.

Governance and Observability

Now we have looked at the advantage and disadvantage of using local models, but our next thing is how do we govern our local models.

The governance is just basically how do we protect our local models, how do we make sure we're regulated by we do things which the government wants us to do.

Software developers, I know we hate doing things the government wants us to do, but we're forced to.

So, how do we run our governance systems on our local models? How do we get the tools to be protected? How do we get other stuff? Yeah, okay.

So, if your organization does a local model deployment, you should always take into account control with artifacts run. That means you should have a model registry. A model registry is where you keep your models. You shouldn't just go to Hugging Face and pick model, try to get your models in a particular registry in your organization, such that you pass some security clearances.

Yeah, so, well, there's a demo, but due to time constraint, anyone who wants the demo, we can watch it after that.

And when it comes to governance, governance and control doesn't only have to do with how we govern our models. Governance and control goes down to what tools we use, how those tools are called. so you have to come with observability of local models. So to govern your local model, make sure they're doing the right thing, you have to observe it.

Well, most software developers use Lansmith. Use Lansmith for observation, observability. Anyone uses Lansmith? Does anyone use Lansmith? What do you use for observability? Like, observe what your agent is doing. If your agent is supposed to call a particular tool, you know that it's calling that tool. So if your IGN is accessing the file, you know it's accessing the file.

Anyone who runs local models, do you do observability? If you don't, you are doing something illegal, because the government will come for you. So please, you try to do observability. It's very important. If you can't, well, you can ask for our service.

Assessing Deployment Maturity

OK, just to be quick, maturity. Before going into deploying of local models, You should know at what level of maturity do you fall in. So you can just put yourself in your maturity level.

Now, this is... Just one minute. Okay.

else.

Smart Routing and Local AI Services

Just to conclude, we have various companies in the UK which offer the service of local model and smart routing.

The smart routing system is, for most companies, which have just, which need local models and frontier models, there's what we call the smart routing system, which you can route to your frontier models and route to your local models, depending on your policies and governance and observability of your systems. So that's where we come to the smart routing system.

I have a sample of a smart routing, anyone who needs to see it in action, we can show that but due to time constraint I'll just keep that for after that so

companies we do that you have the various companies and well mine is just small little not it's nothing so if you just yeah it's something useless there

those are the big guys so if you need you the services of local model deployment in frames optimizations and make your systems faster not just by by hardware implementation, by other stuff.

We can get to all those guys, or you get to us. We do the consultancy and the deployment stuff.

Conclusion

Thank you.

Finished reading?